Kafka Broker filter

The Apache Kafka broker filter decodes the client protocol for Apache Kafka, both the requests and responses in the payload. The message versions in Kafka 2.4.0 are supported. The filter attempts not to influence the communication between client and brokers, so the messages that could not be decoded (due to Kafka client or broker running a newer version than supported by this filter) are forwarded as-is.

  • v2 API reference

  • This filter should be configured with the name envoy.filters.network.kafka_broker.

Attention

The kafka_broker filter is experimental and is currently under active development. Capabilities will be expanded over time and the configuration structures are likely to change.

Configuration

The Kafka Broker filter should be chained with the TCP proxy filter as shown in the configuration snippet below:

  1. listeners:
  2. - address:
  3. socket_address:
  4. address: 127.0.0.1 # Host that Kafka clients should connect to.
  5. port_value: 19092 # Port that Kafka clients should connect to.
  6. filter_chains:
  7. - filters:
  8. - name: envoy.filters.network.kafka_broker
  9. typed_config:
  10. "@type": type.googleapis.com/envoy.config.filter.network.kafka_broker.v2alpha1.KafkaBroker
  11. stat_prefix: exampleprefix
  12. - name: envoy.filters.network.tcp_proxy
  13. typed_config:
  14. "@type": type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
  15. stat_prefix: tcp
  16. cluster: localkafka
  17. clusters:
  18. - name: localkafka
  19. connect_timeout: 0.25s
  20. type: strict_dns
  21. lb_policy: round_robin
  22. hosts:
  23. - socket_address:
  24. address: 127.0.0.1 # Kafka broker's host.
  25. port_value: 9092 # Kafka broker's port.

The Kafka broker needs to advertise the Envoy listener port instead of its own.

  1. # Listener value needs to be equal to cluster value in Envoy config
  2. # (will receive payloads from Envoy).
  3. listeners=PLAINTEXT://127.0.0.1:9092
  4. # Advertised listener value needs to be equal to Envoy's listener
  5. # (will make clients discovering this broker talk to it through Envoy).
  6. advertised.listeners=PLAINTEXT://127.0.0.1:19092

Statistics

Every configured Kafka Broker filter has statistics rooted at kafka.<stat_prefix>., with multiple statistics per message type.

Name

Type

Description

request.TYPE

Counter

Number of times a request of particular type was received from Kafka client

request.unknown

Counter

Number of times a request with format not recognized by this filter was received

request.failure

Counter

Number of times a request with invalid format was received or other processing exception occurred

response.TYPE

Counter

Number of times a response of particular type was received from Kafka broker

response.TYPE_duration

Histogram

Response generation time in milliseconds

response.unknown

Counter

Number of times a response with format not recognized by this filter was received

response.failure

Counter

Number of times a response with invalid format was received or other processing exception occurred