Send data from Tremor to Axiom

This step-by-step guide will help you configure Tremor connectors and events components to interact with your databases, APIs, and ingest data from these sources into Axiom.

Axiom provides a unique way of ingesting Tremor logs into Axiom. With your connector definitions, you can configure Tremor connectors and events components to interact with your external systems, such as databases, message queues, or APIs, and eventually ingest data from these sources into Axiom.

Installation

To install tremor grab the latest package from the runtime releases tag, and install it on your local machine.

Configuration using HTTP

To send logs via Tremor to Axiom, you need to create a configuration file. For example, create axiom-http.troy with the following content (using a file as example data source):

define flow client_sink_only
flow
  use std::time::nanos;
  use tremor::pipelines;

  define connector input from file
  args
    file = "in.json"  # Default input file is 'in.json' in current working directory
  with
    codec = "json",   # Data is JSON encoded
    preprocessors = ["separate"],   # Data is newline separated
    config = {
        "path": args.file,
        "mode": "read"
    },
  end;
  create connector input;

  define connector http_client from http_client
  args
    dataset,
    token
  with
    config = {
      "url": "https://api.axiom.co/v1/datasets/#{args.dataset}/ingest",
      "tls": true,
      "method": "POST",
      "headers": {
        "Authorization": "Bearer #{args.token}"
      },
      "timeout": nanos::from_seconds(10),
      "mime_mapping": {
        "*/*": {"name": "json"}, 
      } 
    }
  end;
  create connector http_client
  with
    dataset = "$DATASET_NAME",
    token = "$API_TOKEN"
  end;

  create pipeline passthrough from pipelines::passthrough;

  connect /connector/input to /pipeline/passthrough;
  connect /pipeline/passthrough to /connector/http_client;

end;

deploy flow client_sink_only;

This assumes you have set TREMOR_PATH in your environment pointing to tremor-runtime/tremor-script/lib if you are using a src clone then you can execute it as follows tremor server run axiom-http.troy

The$DATASET_NAME dataset you want to send logs to in Axiom, and the $API_TOKEN is your Axiom API token for ingesting and quering your Tremor logs.

Configuration using Syslog

You can also send logs via Tremor to the Syslog endpoint using a file as an example data source.

1. Click Settings > Endpoints.
2. Click New endpoint.
3. Click .
4. Name the endpoint.
5. Select the dataset where you want to send data.
6. Copy the URL displayed for the newly created endpoint. This is the target URL where you send the data.

In the code below, replace url with the URL of your Syslog endpoint.

define flow client_sink_only
flow
  use std::time::nanos;
  use tremor::pipelines;

  define connector input from file
  args
    file = "in.json"  # Default input file is 'in.json' in current working directory
  with
    codec = "json",    # Data is JSON encoded
    preprocessors = ["separate"],    # Data is newline separated
    config = {
        "path": args.file,
        "mode": "read"
    },
  end;
  create connector input;

define connector syslog_forwarder from tcp_client
args
  endpoint_hostport,
with
    tls = true,
    codec = "syslog",
    config = {
      "url": "#{args.endpoint_hostport}",
      "no_delay": false,
      "buf_size": 1024,
    },
    reconnect = {
      "retry": {
        "interval_ms": 100,
        "growth_rate": 2,
        "max_retries": 3,
      }
    }
 end;
  create connector syslog_forwarder 
  with
    endpoint_hostport = "tcp+tls://testsyslog.syslog.axiom.co:6514"
  end;

  create pipeline passthrough from pipelines::passthrough;

  connect /connector/input to /pipeline/passthrough;
  connect /pipeline/passthrough to /connector/syslog_forwarder;

end;

deploy flow client_sink_only;