Fluentd

Loki has a Fluentd output plugin calledfluent-plugin-grafana-loki that enables shipping logs to a private Lokiinstance or Grafana Cloud.

The plugin offers two line formats and uses protobuf to send compressed data toLoki.

Key features:

• extra_labels: Labels to be added to every line of a log file, useful fordesignating environments
• label_keys: Customizable list of keys for stream labels
• line_format: Format to use when flattening the record to a log line (json or key_value).

Installation

$ gem install fluent-plugin-grafana-loki

Usage

In your Fluentd configuration, use @type loki. Additional configuration isoptional. Default values look like this:

<match **>
  @type loki
  url "https://logs-us-west1.grafana.net"
  username "#{ENV['LOKI_USERNAME']}"
  password "#{ENV['LOKI_PASSWORD']}"
  extra_labels {"env":"dev"}
  flush_interval 10s
  flush_at_shutdown true
  buffer_chunk_limit 1m
</match>

Multi-worker usage

Loki doesn’t currently support out-of-order inserts - if you try to insert a logentry with an earlier timestamp after a log entry with with identical labels buta later timestamp, the insert will fail with the messageHTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order. Therefore, in order to use this plugin in a multi-worker Fluentd setup,you’ll need to include the worker ID in the labels sent to Loki.

For example, usingfluent-plugin-record-modifier:

<filter mytag>
    @type record_modifier
    <record>
        fluentd_worker "#{worker_id}"
    </record>
</filter>
<match mytag>
  @type loki
  # ...
  label_keys "fluentd_worker"
  # ...
</match>

Docker Image

There is a Docker image grafana/fluent-plugin-grafana-loki:master whichcontains default configuration files to git log informationa host’s /var/log directory, and from the host’s journald. To use it, you can setthe LOKI_URL, LOKI_USERNAME, and LOKI_PASSWORD environment variables(LOKI_USERNAME and LOKI_PASSWORD can be left blank if Loki is not protectedbehind an authenticating proxy).

An example Docker Swarm Compose configuration looks like:

services:
  fluentd:
    image: grafana/fluent-plugin-grafana-loki:master
    command:
      - "fluentd"
      - "-v"
      - "-p"
      - "/fluentd/plugins"
    environment:
      LOKI_URL: http://loki:3100
      LOKI_USERNAME:
      LOKI_PASSWORD:
    deploy:
      mode: global
    configs:
      - source: loki_config
        target: /fluentd/etc/loki/loki.conf
    networks:
      - loki
    volumes:
      - host_logs:/var/log
      # Needed for journald log ingestion:
      - /etc/machine-id:/etc/machine-id
      - /dev/log:/dev/log
      - /var/run/systemd/journal/:/var/run/systemd/journal/
    logging:
      options:
         tag: infra.monitoring

Configuration

Proxy Support

Starting with version 0.8.0, this gem uses excon, which supports proxy withenvironment variables - https://github.com/excon/excon#proxy-support

url

The URL of the Loki server to send logs to. When sending data the publish path(/loki/api/v1/push) will automatically be appended. By default the URL is set tohttps://logs-us-west1.grafana.net, the URL of the Grafana Labs hostedLoki service.

username / password

Specify a username and password if the Loki server requires authentication.If using the Grafana Labs’ hosted Loki, the username needs to be set to yourinstanceId and the password should be a grafana.com API Key.

tenant

Loki is a multi-tenant log storage platform and all requests sent must include atenant. For some installations (like Hosted Loki) the tenant will be setautomatically by an authenticating proxy. Otherwise you can define a tenant tobe passed through. The tenant can be any string value.

output format

Loki is intended to index and group log streams using only a small set oflabels and is not intended for full-text indexing. When sending logs to Loki,the majority of log message will be sent as a single log “line”.

There are few configurations settings to control the output format:

• extra_labels: (default: nil) set of labels to include with every Lokistream. (e.g., {"env":"dev", "datacenter": "dc1"})

• remove_keys: (default: nil) comma separated list of record keys toremove. All other keys will be placed into the log line.

• label_keys: (default: “job,instance”) comma separated list of keys to use asstream labels.

• line_format: format to use when flattening the record to a log line. Validvalues are json or key_value. If set to json the log line sent to Lokiwill be the fluentd record (excluding any keys extracted out as labels) dumpedas json. If set to key_value, the log line will be each item in the recordconcatenated together (separated by a single space) in the format<key>=<value>.

• drop_single_key: if set to true, when the set of extracted label_keys after dropping with remove_keys, the log line sent to Loki will just be the value of the single remaining record key.

Buffer options

fluentd-plugin-loki extends Fluentd’s builtin Outputplugin and usesthe compat_parameters plugin helper. It adds the following options:

buffer_type memory
flush_interval 10s
retry_limit 17
retry_wait 1.0
num_threads 1