Parser

The Parser Filter plugin allows for parsing fields in event records.

Configuration Parameters

The plugin supports the following configuration parameters:

Key Description Default
Key_Name Specify field name in record to parse.
Parser Specify the parser name to interpret the field. Multiple Parser entries are allowed (one per line).
Preserve_Key Keep original Key_Name field in the parsed result. If false, the field will be removed. False
Reserve_Data Keep all other original fields in the parsed result. If false, all other original fields will be removed. False
Unescape_Key If the key is an escaped string (e.g: stringify JSON), unescape the string before applying the parser. False

Getting Started

Configuration File

This is an example of parsing a record {"data":"100 0.5 true This is example"}.

The plugin needs a parser file which defines how to parse each field.

  1. [PARSER]
  2. Name dummy_test
  3. Format regex
  4. Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$

The path of the parser file should be written in configuration file under the [SERVICE] section.

  1. [SERVICE]
  2. Parsers_File /path/to/parsers.conf
  3. [INPUT]
  4. Name dummy
  5. Tag dummy.data
  6. Dummy {"data":"100 0.5 true This is example"}
  7. [FILTER]
  8. Name parser
  9. Match dummy.*
  10. Key_Name data
  11. Parser dummy_test
  12. [OUTPUT]
  13. Name stdout
  14. Match *

The output is

  1. $ fluent-bit -c dummy.conf
  2. Fluent Bit v1.x.x
  3. * Copyright (C) 2019-2020 The Fluent Bit Authors
  4. * Copyright (C) 2015-2018 Treasure Data
  5. * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  6. * https://fluentbit.io
  7. [2017/07/06 22:33:12] [ info] [engine] started
  8. [0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  9. [1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  10. [2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  11. [3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]

You can see the records {"data":"100 0.5 true This is example"} are parsed.

Preserve original fields

By default, the parser plugin only keeps the parsed fields in its output.

If you enable Reserve_Data, all other fields are preserved:

  1. [PARSER]
  2. Name dummy_test
  3. Format regex
  4. Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
  1. [SERVICE]
  2. Parsers_File /path/to/parsers.conf
  3. [INPUT]
  4. Name dummy
  5. Tag dummy.data
  6. Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
  7. [FILTER]
  8. Name parser
  9. Match dummy.*
  10. Key_Name data
  11. Parser dummy_test
  12. Reserve_Data On

This will produce the output:

  1. $ fluent-bit -c dummy.conf
  2. Fluent-Bit v0.12.0
  3. Copyright (C) Treasure Data
  4. [2017/07/06 22:33:12] [ info] [engine] started
  5. [0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
  6. [1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
  7. [2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
  8. [3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]

If you enable Reserved_Data and Preserve_Key, the original key field will be preserved as well:

  1. [PARSER]
  2. Name dummy_test
  3. Format regex
  4. Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$
  1. [SERVICE]
  2. Parsers_File /path/to/parsers.conf
  3. [INPUT]
  4. Name dummy
  5. Tag dummy.data
  6. Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
  7. [FILTER]
  8. Name parser
  9. Match dummy.*
  10. Key_Name data
  11. Parser dummy_test
  12. Reserve_Data On
  13. Preserve_Key On

This will produce the following output:

  1. $ fluent-bit -c dummy.conf
  2. Fluent-Bit v0.12.0
  3. Copyright (C) Treasure Data
  4. [2017/07/06 22:33:12] [ info] [engine] started
  5. [0] dummy.data: [1499347993.001371317, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  6. [1] dummy.data: [1499347994.001303118, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  7. [2] dummy.data: [1499347995.001296133, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
  8. [3] dummy.data: [1499347996.001320284, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]