Configuration Filters

Flume provides a tool for injecting sensitive or generated data into the configurationin the form of configuration filters. A configuration key can be set as the value of configuration propertiesand it will be replaced by the configuration filter with the value it represents.

Common usage of config filters

The format is similar to the Java Expression Language, howeverit is currently not a fully working EL expression parser, just a format that looks like it.

  1. <agent_name>.configfilters = <filter_name>
  2. <agent_name>.configfilters.<filter_name>.type = <filter_type>
  3.  
  4. <agent_name>.sources.<source_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
  5. <agent_name>.sinks.<sink_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
  6. <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
  7. #or
  8. <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>["<key_for_sensitive_or_generated_data>"]}
  9. #or
  10. <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>[<key_for_sensitive_or_generated_data>]}
  11. #or
  12. <agent_name>.<component_type>.<component_name>.parameter = some_constant_data${<filter_name>[<key_for_sensitive_or_generated_data>]}

Environment Variable Config Filter

Property NameDefaultDescription
type–The component type name has to be env

Example

To hide a password in the configuration set its value as in the following example.

  1. a1.sources = r1
  2. a1.channels = c1
  3. a1.configfilters = f1
  4.  
  5. a1.configfilters.f1.type = env
  6.  
  7. a1.sources.r1.channels =  c1
  8. a1.sources.r1.type = http
  9. a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value Secret123

Here the a1.sources.r1.keystorePassword configuration property will get the value of the my_keystore_passwordenvironment variable. One way to set the environment variable is to run flume agent like this:

$ my_keystore_password=Secret123 bin/flume-ng agent —conf conf —conf-file example.conf …

External Process Config Filter

Property NameDefaultDescription
type–The component type name has to be external
command–The command that will be executed to get the value for the given key. The command will be called like: <command> <key> And expected to return a single line value with exit code 0.
charsetUTF-8The characterset of the returned string.

Example

To hide a password in the configuration set its value as in the following example.

  1. a1.sources = r1
  2. a1.channels = c1
  3. a1.configfilters = f1
  4.  
  5. a1.configfilters.f1.type = external
  6. a1.configfilters.f1.command = /usr/bin/passwordResolver.sh
  7. a1.configfilters.f1.charset = UTF-8
  8.  
  9. a1.sources.r1.channels =  c1
  10. a1.sources.r1.type = http
  11. a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value Secret123

In this example flume will run the following command to get the value

$ /usr/bin/passwordResolver.sh my_keystore_password

The passwordResolver.sh will return Secret123 with an exit code 0.

Example 2

To generate a part of the directory for rolling file sink set its value as in the following example.

  1. a1.sources = r1
  2. a1.channels = c1
  3. a1.configfilters = f1
  4.  
  5. a1.configfilters.f1.type = external
  6. a1.configfilters.f1.command = /usr/bin/generateUniqId.sh
  7. a1.configfilters.f1.charset = UTF-8
  8.  
  9. a1.sinks = k1
  10. a1.sinks.k1.type = file_roll
  11. a1.sinks.k1.channel = c1
  12. a1.sinks.k1.sink.directory = /var/log/flume/agent_${f1['agent_name']} # will be /var/log/flume/agent_1234

In this example flume will run the following command to get the value

$ /usr/bin/generateUniqId.sh agent_name

The generateUniqId.sh will return 1234 with an exit code 0.

Hadoop Credential Store Config Filter

A hadoop-common library needed on the classpath for this feature (2.6+ version).If hadoop is installed the agent adds it to the classpath automatically

Property NameDefaultDescription
type–The component type name has to be hadoop
credential.provider.path–The provider path. See hadoop documentation _here: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Configuring_the_Provider_Path
credstore.java-keystore-provider.password-file–The name of the password file if a file is used to store the password. The file must e on the classpath.Provider password can be set with the HADOOP_CREDSTORE_PASSWORD environment variable or left empty.

Example

To hide a password in the configuration set its value as in the following example.

  1. a1.sources = r1
  2. a1.channels = c1
  3. a1.configfilters = f1
  4.  
  5. a1.configfilters.f1.type = hadoop
  6. a1.configfilters.f1.credential.provider.path = jceks://file/<path_to_jceks file>
  7.  
  8. a1.sources.r1.channels =  c1
  9. a1.sources.r1.type = http
  10. a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value from the credential store