Grep

The Grep Filter plugin allows to match or exclude specific records based in regular expression patterns.

Configuration Parameters

The plugin supports the following configuration parameters:

Key Value Format Description
Regex FIELD REGEX Keep records which field matches the regular expression.
Exclude FIELD REGEX Exclude records which field matches the regular expression.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following example assumes that you have a file called lines.txt with the following content

  1. aaa
  2. aab
  3. bbb
  4. ccc
  5. ddd
  6. eee
  7. fff
  8. ggg

Command Line

Note: using the command line mode need special attention to quote the regular expressions properly. It’s suggested to use a configuration file.

The following command will load the tail plugin and read the content of lines.txt file. Then the grep filter will apply a regular expression rule over the log field (created by tail plugin) and only pass the records which field value starts with aa:

  1. $ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout

Configuration File

  1. [INPUT]
  2.     Name   tail
  3.     Path   lines.txt
  5. [FILTER]
  6.     Name   grep
  7.     Match  *
  8.     Regex  log aa
  10. [OUTPUT]
  11.     Name   stdout
  12.     Match  *

The filter allows to use multiple rules which are applied in order, you can have many Regex and Exclude entries as required.

Nested fields example

Currently nested fields are not supported. If you have records in the following format

  1. {
  2.     "kubernetes": {
  3.         "pod_name": "myapp-0",
  4.         "namespace_name": "default",
  5.         "pod_id": "216cd7ae-1c7e-11e8-bb40-000c298df552",
  6.         "labels": {
  7.             "app": "myapp"
  8.         },
  9.         "host": "minikube",
  10.         "container_name": "myapp",
  11.         "docker_id": "370face382c7603fdd309d8c6aaaf434fd98b92421ce7c7c8aafe7697d4aa362"
  12.     }
  13. }

and if you want to exclude records that match given nested field (for example kubernetes.labels.app), you could use combination of nest and grep filters. Here is an example that will exclude records that match kubernetes.labels.app: myapp:

  1. [FILTER]
  2.     Name         nest
  3.     Match        *
  4.     Operation    lift
  5.     Nested_under kubernetes
  7. [FILTER]
  8.     Name         nest
  9.     Match        *
  10.     Operation    lift
  11.     Nested_under labels
  13. [FILTER]
  14.     Name    grep
  15.     Match   *
  16.     Exclude app myapp