Apache Kafka examples

The following examples will help you understand how to use the different Apache Kafka components for Knative.

Prerequisites

All examples require:

  • A Kubernetes cluster with
  • Knative Eventing v0.9+
  • Knative Serving v0.9+
  • An Apache Kafka cluster

Setting up Apache Kafka

If you want to run the Apache Kafka cluster on Kubernetes, the simplest option is to install it by using Strimzi.

  1. Create a namespace for your Apache Kafka installation, like kafka:

    1. kubectl create namespace kafka
  2. Install the Strimzi operator, like:

    1. curl -L "https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.2/strimzi-cluster-operator-0.16.2.yaml" \
    2. | sed 's/namespace: .*/namespace: kafka/' \
    3. | kubectl -n kafka apply -f -
  3. Describe the size of your Apache Kafka installation in kafka.yaml, like:

    1. apiVersion: kafka.strimzi.io/v1beta2
    2. kind: Kafka
    3. metadata:
    4. name: my-cluster
    5. spec:
    6. kafka:
    7. version: 2.4.0
    8. replicas: 1
    9. listeners:
    10. plain: {}
    11. tls: {}
    12. config:
    13. offsets.topic.replication.factor: 1
    14. transaction.state.log.replication.factor: 1
    15. transaction.state.log.min.isr: 1
    16. log.message.format.version: "2.4"
    17. storage:
    18. type: ephemeral
    19. zookeeper:
    20. replicas: 3
    21. storage:
    22. type: ephemeral
    23. entityOperator:
    24. topicOperator: {}
    25. userOperator: {}
  4. Deploy the Apache Kafka cluster

    1. $ kubectl apply -n kafka -f kafka.yaml

This will install a small, non-production, cluster of Apache Kafka. To verify your installation, check if the pods for Strimzi are all up, in the kafka namespace:

  1. $ kubectl get pods -n kafka
  2. NAME READY STATUS RESTARTS AGE
  3. my-cluster-entity-operator-65995cf856-ld2zp 3/3 Running 0 102s
  4. my-cluster-kafka-0 2/2 Running 0 2m8s
  5. my-cluster-zookeeper-0 2/2 Running 0 2m39s
  6. my-cluster-zookeeper-1 2/2 Running 0 2m49s
  7. my-cluster-zookeeper-2 2/2 Running 0 2m59s
  8. strimzi-cluster-operator-77555d4b69-sbrt4 1/1 Running 0 3m14s

NOTE: For production ready installs check Strimzi.

Installation script

If you want to install the latest version of Strimzi, in just one step, we have a script for your convenience, which does exactly the same steps that are listed above:

  1. $ ./kafka_setup.sh

Examples of Apache Kafka and Knative

A number of different examples, showing the KafkaSource, KafkaChannel and KafkaBinding can be found here: