我的书签
添加书签
移除书签Flume Sink Processors
Sink groups allow users to group multiple sinks into one entity.Sink processors can be used to provide load balancing capabilities over allsinks inside the group or to achieve fail over from one sink to another incase of temporal failure.
Required properties are in bold.
| Property Name | Default | Description |
|---|---|---|
| sinks | – | Space-separated list of sinks that are participating in the group |
| processor.type | default | The component type name, needs to be default, failover or load_balance |
Example for agent named a1:
- a1.sinkgroups = g1
- a1.sinkgroups.g1.sinks = k1 k2
- a1.sinkgroups.g1.processor.type = load_balance
Default Sink Processor
Default sink processor accepts only a single sink. User is not forcedto create processor (sink group) for single sinks. Instead user can followthe source - channel - sink pattern that was explained above in this userguide.
Failover Sink Processor
Failover Sink Processor maintains a prioritized list of sinks, guaranteeingthat so long as one is available events will be processed (delivered).
The failover mechanism works by relegating failed sinks to a pool wherethey are assigned a cool down period, increasing with sequential failuresbefore they are retried. Once a sink successfully sends an event, it isrestored to the live pool. The Sinks have a priority associated with them,larger the number, higher the priority. If a Sink fails while sending a Eventthe next Sink with highest priority shall be tried next for sending Events.For example, a sink with priority 100 is activated before the Sink with priority80. If no priority is specified, thr priority is determined based on the order in whichthe Sinks are specified in configuration.
To configure, set a sink groups processor to failover and setpriorities for all individual sinks. All specified priorities mustbe unique. Furthermore, upper limit to failover time can be set(in milliseconds) using maxpenalty property.
Required properties are in bold.
| Property Name | Default | Description |
|---|---|---|
| sinks | – | Space-separated list of sinks that are participating in the group |
| processor.type | default | The component type name, needs to be failover |
| processor.priority.<sinkName> | – | Priority value. <sinkName> must be one of the sink instances associated with the current sink groupA higher priority value Sink gets activated earlier. A larger absolute value indicates higher priority |
| processor.maxpenalty | 30000 | The maximum backoff period for the failed Sink (in millis) |
Example for agent named a1:
- a1.sinkgroups = g1
- a1.sinkgroups.g1.sinks = k1 k2
- a1.sinkgroups.g1.processor.type = failover
- a1.sinkgroups.g1.processor.priority.k1 = 5
- a1.sinkgroups.g1.processor.priority.k2 = 10
- a1.sinkgroups.g1.processor.maxpenalty = 10000
Load balancing Sink Processor
Load balancing sink processor provides the ability to load-balance flow overmultiple sinks. It maintains an indexed list of active sinks on which theload must be distributed. Implementation supports distributing load usingeither via round_robin or random selection mechanisms.The choice of selection mechanism defaults to round_robin type,but can be overridden via configuration. Custom selection mechanisms aresupported via custom classes that inherits from AbstractSinkSelector.
When invoked, this selector picks the next sink using its configured selectionmechanism and invokes it. For round_robin and random In case the selected sinkfails to deliver the event, the processor picks the next available sink viaits configured selection mechanism. This implementation does not blacklistthe failing sink and instead continues to optimistically attempt everyavailable sink. If all sinks invocations result in failure, the selectorpropagates the failure to the sink runner.
If backoff is enabled, the sink processor will blacklistsinks that fail, removing them for selection for a given timeout. When thetimeout ends, if the sink is still unresponsive timeout is increasedexponentially to avoid potentially getting stuck in long waits on unresponsivesinks. With this disabled, in round-robin all the failed sinks load will bepassed to the next sink in line and thus not evenly balanced
Required properties are in bold.
| Property Name | Default | Description |
|---|---|---|
| processor.sinks | – | Space-separated list of sinks that are participating in the group |
| processor.type | default | The component type name, needs to be load_balance |
| processor.backoff | false | Should failed sinks be backed off exponentially. |
| processor.selector | round_robin | Selection mechanism. Must be either round_robin, randomor FQCN of custom class that inherits from AbstractSinkSelector |
| processor.selector.maxTimeOut | 30000 | Used by backoff selectors to limit exponential backoff (in milliseconds) |
Example for agent named a1:
- a1.sinkgroups = g1
- a1.sinkgroups.g1.sinks = k1 k2
- a1.sinkgroups.g1.processor.type = load_balance
- a1.sinkgroups.g1.processor.backoff = true
- a1.sinkgroups.g1.processor.selector = random
Custom Sink Processor
Custom sink processors are not supported at the moment.
