Apache Kafka Broker

The Apache Kafka Broker is a native Broker implementation, that reduces network hops, supports any Kafka version, and has a better integration with Apache Kafka for the Knative Broker and Trigger model.

Notable features are:

Prerequisites

Knative Eventing installation.

Installation

  1. Install the Kafka broker by entering the following command:

    1. kubectl apply --filename https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/v0.17.0/eventing-kafka-broker.yaml
  2. Verify that kafka-broker-controller, kafka-broker-receiver and kafka-broker-dispatcher are running, by entering the following command:

    1. kubectl get deployments.apps -n knative-eventing

    Example output:

    1. NAME READY UP-TO-DATE AVAILABLE AGE
    2. eventing-controller 1/1 1 1 10s
    3. eventing-webhook 1/1 1 1 9s
    4. kafka-broker-controller 1/1 1 1 3s
    5. kafka-broker-dispatcher 1/1 1 1 4s
    6. kafka-broker-receiver 1/1 1 1 5s

Create a Kafka Broker

A Kafka Broker object looks like this:

  1. apiVersion: eventing.knative.dev/v1
  2. kind: Broker
  3. metadata:
  4. annotations:
  5. # case-sensitive
  6. eventing.knative.dev/broker.class: Kafka
  7. name: default
  8. namespace: default
  9. spec:
  10. # Configuration specific to this broker.
  11. config:
  12. apiVersion: v1
  13. kind: ConfigMap
  14. name: kafka-broker-config
  15. namespace: knative-eventing
  16. # Optional dead letter sink, you can specify either:
  17. # - deadLetterSink.ref, which is a reference to a Callable
  18. # - deadLetterSink.uri, which is an absolute URI to a Callable (It can potentially be out of the Kubernetes cluster)
  19. delivery:
  20. deadLetterSink:
  21. ref:
  22. apiVersion: serving.knative.dev/v1
  23. kind: Service
  24. name: dlq-service

spec.config should reference any ConfigMap that looks like the following:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: kafka-broker-config
  5. namespace: knative-eventing
  6. data:
  7. # Number of topic partitions
  8. default.topic.partitions: "10"
  9. # Replication factor of topic messages.
  10. default.topic.replication.factor: "1"
  11. # A comma separated list of bootstrap servers. (It can be in or out the k8s cluster)
  12. bootstrap.servers: "my-cluster-kafka-bootstrap.kafka:9092"

The above ConfigMap is installed in the cluster. You can edit the configuration or create a new one with the same values depending on your needs.

Set as default broker implementation

To set the Kafka broker as the default implementation for all brokers in the Knative deployment, you can apply global settings by modifying the config-br-defaults ConfigMap in the knative-eventing namespace.

This allows you to avoid configuring individual or per-namespace settings for each broker, such as metadata.annotations.eventing.knative.dev/broker.class or spec.config.

The following YAML is an example of a config-br-defaults ConfigMap using Kafka broker as the default implementation.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: config-br-defaults
  5. namespace: knative-eventing
  6. data:
  7. default-br-config: |
  8. clusterDefault:
  9. brokerClass: Kafka
  10. apiVersion: v1
  11. kind: ConfigMap
  12. name: kafka-broker-config
  13. namespace: knative-eventing
  14. namespaceDefaults:
  15. namespace1:
  16. brokerClass: Kafka
  17. apiVersion: v1
  18. kind: ConfigMap
  19. name: kafka-broker-config
  20. namespace: knative-eventing
  21. namespace2:
  22. brokerClass: Kafka
  23. apiVersion: v1
  24. kind: ConfigMap
  25. name: kafka-broker-config
  26. namespace: knative-eventing

Kafka Producer and Consumer configurations

Knative exposes all available Kafka producer and consumer configurations that can be modified to suit your workloads.

You can change these configurations by modifying the config-kafka-broker-data-plane ConfigMap in the knative-eventing namespace.

Documentation for the settings available in this ConfigMap is available on the Apache Kafka website, in particular, Producer configurations and Consumer configurations.

Enable debug logging for data plane components

The following YAML shows the default logging configuration for data plane components, that is created during the installation step:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: kafka-broker-config-logging
  5. namespace: knative-eventing
  6. data:
  7. config.xml: |
  8. <configuration>
  9. <appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
  10. <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
  11. </appender>
  12. <root level="INFO">
  13. <appender-ref ref="jsonConsoleAppender"/>
  14. </root>
  15. </configuration>

To change the logging level to DEBUG, you need to:

  1. Apply the following kafka-broker-config-logging ConfigMap or replace level="INFO" with level="DEBUG" to the ConfigMap kafka-broker-config-logging:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4. name: kafka-broker-config-logging
    5. namespace: knative-eventing
    6. data:
    7. config.xml: |
    8. <configuration>
    9. <appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
    10. <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    11. </appender>
    12. <root level="DEBUG">
    13. <appender-ref ref="jsonConsoleAppender"/>
    14. </root>
    15. </configuration>
  2. Restart the kafka-broker-receiver and the kafka-broker-dispatcher, by entering the following commands:

    1. kubectl rollout restart deployment -n knative-eventing kafka-broker-receiver
    2. kubectl rollout restart deployment -n knative-eventing kafka-broker-dispatcher

Additional information