description: A full feature set to access content of your records

Record Accessor

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.

Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.

consider Record Accessor a simple grammar to specify record content and other miscellaneous values.

Format

A record accessor rule starts with the character $. Using the structured content above as an example the following table describes how to access a record:

  1. {
  2.   "log": "some message",
  3.   "stream": "stdout",
  4.   "labels": {
  5.      "color": "blue", 
  6.      "unset": null,
  7.      "project": {
  8.          "env": "production"
  9.       }
  10.   }
  11. }

The following table describe some accessing rules and the expected returned value:

Format Accessed Value
$log “some message”
$labels[‘color’] “blue”
$labels[‘project’][‘env’] “production”
$labels[‘unset’] null
$labels[‘undefined’]

If the accessor key does not exist in the record like the last example $labels['undefined'] , the operation is simply omitted, no exception will occur.

Usage Example

The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using grep that only matches where labels have a color blue:

  1. [SERVICE]
  2.     flush        1
  3.     log_level    info
  4.     parsers_file parsers.conf
  6. [INPUT]
  7.     name      tail
  8.     path      test.log
  9.     parser    json
  11. [FILTER]
  12.     name      grep
  13.     match     *
  14.     regex     $labels['color'] ^blue$
  16. [OUTPUT]
  17.     name      stdout
  18.     match     *
  19.     format    json_lines

The file content to process in test.log is the following:

  1. {"log": "message 1", "labels": {"color": "blue"}}
  2. {"log": "message 2", "labels": {"color": "red"}}
  3. {"log": "message 3", "labels": {"color": "green"}}
  4. {"log": "message 4", "labels": {"color": "blue"}}

Running Fluent Bit with the configuration above the output will be:

  1. $ bin/fluent-bit -c fluent-bit.conf 
  2. Fluent Bit v1.x.x
  3. * Copyright (C) 2019-2020 The Fluent Bit Authors
  4. * Copyright (C) 2015-2018 Treasure Data
  5. * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  6. * https://fluentbit.io
  8. [2020/09/11 16:11:07] [ info] [engine] started (pid=1094177)
  9. [2020/09/11 16:11:07] [ info] [storage] version=1.0.5, initializing...
  10. [2020/09/11 16:11:07] [ info] [storage] in-memory
  11. [2020/09/11 16:11:07] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
  12. [2020/09/11 16:11:07] [ info] [sp] stream processor started
  13. [2020/09/11 16:11:07] [ info] inotify_fs_add(): inode=55716713 watch_fd=1 name=test.log
  14. {"date":1599862267.483684,"log":"message 1","labels":{"color":"blue"}}
  15. {"date":1599862267.483692,"log":"message 4","labels":{"color":"blue"}}