Quarkus - Using Apache Kafka with Reactive Messaging

This guide demonstrates how your Quarkus application can utilize MicroProfile Reactive Messaging to interact with Apache Kafka.

Prerequisites

To complete this guide, you need:

  • less than 15 minutes

  • an IDE

  • JDK 1.8+ installed with JAVA_HOME configured appropriately

  • Apache Maven 3.5.3+

  • A running Kafka cluster, or Docker Compose to start a development cluster

  • GraalVM installed if you want to run in native mode.

Architecture

In this guide, we are going to generate (random) prices in one component.These prices are written in a Kafka topic (prices).A second component reads from the prices Kafka topic and apply some magic conversion to the price.The result is sent to an in-memory stream consumed by a JAX-RS resource.The data is sent to a browser using server-sent events.

Architecture

Solution

We recommend that you follow the instructions in the next sections and create the application step by step.However, you can go right to the completed example.

Clone the Git repository: git clone https://github.com/quarkusio/quarkus-quickstarts.git, or download an archive.

The solution is located in the kafka-quickstart directory.

Creating the Maven Project

First, we need a new project. Create a new project with the following command:

  1. mvn io.quarkus:quarkus-maven-plugin:1.0.0.CR1:create \
  2.     -DprojectGroupId=org.acme \
  3.     -DprojectArtifactId=kafka-quickstart \
  4.     -Dextensions="kafka"
  5. cd kafka-quickstart

This command generates a Maven project, importing the Reactive Messaging and Kafka connector extensions.

Starting Kafka

Then, we need a Kafka cluster.You can follow the instructions from the Apache Kafka web site or create a docker-compose.yaml file with the following content:

  1. version: '2'
  3. services:
  5.   zookeeper:
  6.     image: strimzi/kafka:0.11.3-kafka-2.1.0
  7.     command: [
  8.       "sh", "-c",
  9.       "bin/zookeeper-server-start.sh config/zookeeper.properties"
  10.     ]
  11.     ports:
  12.       - "2181:2181"
  13.     environment:
  14.       LOG_DIR: /tmp/logs
  16.   kafka:
  17.     image: strimzi/kafka:0.11.3-kafka-2.1.0
  18.     command: [
  19.       "sh", "-c",
  20.       "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
  21.     ]
  22.     depends_on:
  23.       - zookeeper
  24.     ports:
  25.       - "9092:9092"
  26.     environment:
  27.       LOG_DIR: "/tmp/logs"
  28.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
  29.       KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
  30.       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Once created, run docker-compose up.

This is a development cluster, do not use in production.

The price generator

Create the src/main/java/org/acme/quarkus/sample/PriceGenerator.java file, with the following content:

  1. package org.acme.quarkus.sample;
  3. import io.reactivex.Flowable;
  4. import org.eclipse.microprofile.reactive.messaging.Outgoing;
  6. import javax.enterprise.context.ApplicationScoped;
  7. import java.util.Random;
  8. import java.util.concurrent.TimeUnit;
  10. /**
  11.  * A bean producing random prices every 5 seconds.
  12.  * The prices are written to a Kafka topic (prices). The Kafka configuration is specified in the application configuration.
  13.  */
  14. @ApplicationScoped
  15. public class PriceGenerator {
  17.     private Random random = new Random();
  19.     @Outgoing("generated-price")                        (1)
  20.     public Flowable<Integer> generate() {               (2)
  21.         return Flowable.interval(5, TimeUnit.SECONDS)
  22.                 .map(tick -> random.nextInt(100));
  23.     }
  25. }
1Instruct Reactive Messaging to dispatch the items from returned stream to generated-price.
2The method returns a RX Java 2 stream (Flowable) emitting a random price every 5 seconds.

The method returns a Reactive Stream. The generated items are sent to the stream named generated-price.This stream is mapped to Kafka using the application.properties file that we will create soon.

The price converter

The price converter reads the prices from Kafka, and transforms them.Create the src/main/java/org/acme/quarkus/sample/PriceConverter.java file with the following content:

  1. package org.acme.quarkus.sample;
  3. import io.smallrye.reactive.messaging.annotations.Broadcast;
  4. import org.eclipse.microprofile.reactive.messaging.Incoming;
  5. import org.eclipse.microprofile.reactive.messaging.Outgoing;
  7. import javax.enterprise.context.ApplicationScoped;
  9. /**
  10.  * A bean consuming data from the "prices" Kafka topic and applying some conversion.
  11.  * The result is pushed to the "my-data-stream" stream which is an in-memory stream.
  12.  */
  13. @ApplicationScoped
  14. public class PriceConverter {
  16.     private static final double CONVERSION_RATE = 0.88;
  18.     @Incoming("prices")                                 (1)
  19.     @Outgoing("my-data-stream")                         (2)
  20.     @Broadcast                                          (3)
  21.     public double process(int priceInUsd) {
  22.         return priceInUsd * CONVERSION_RATE;
  23.     }
  25. }
1Indicates that the method consumes the items from the prices topic
2Indicates that the objects returned by the method are sent to the my-data-stream stream
3Indicates that the item are dispatched to all subscribers

The process method is called for every Kafka record from the prices topic (configured in the application configuration).Every result is sent to the my-data-stream in-memory stream.

The price resource

Finally, let’s bind our stream to a JAX-RS resource.Creates the src/main/java/org/acme/quarkus/sample/PriceResource.java file with the following content:

  1. package org.acme.quarkus.sample;
  3. import io.smallrye.reactive.messaging.annotations.Channel;
  4. import org.reactivestreams.Publisher;
  6. import javax.inject.Inject;
  7. import javax.ws.rs.GET;
  8. import javax.ws.rs.Path;
  9. import javax.ws.rs.Produces;
  10. import javax.ws.rs.core.MediaType;
  11. import org.jboss.resteasy.annotations.SseElementType;
  13. /**
  14.  * A simple resource retrieving the in-memory "my-data-stream" and sending the items as server-sent events.
  15.  */
  16. @Path("/prices")
  17. public class PriceResource {
  19.     @Inject
  20.     @Channel("my-data-stream") Publisher<Double> prices; (1)
  22.     @GET
  23.     @Path("/stream")
  24.     @Produces(MediaType.SERVER_SENT_EVENTS) (2)
  25.     @SseElementType("text/plain") (3)
  26.     public Publisher<Double> stream() { (4)
  27.         return prices;
  28.     }
  29. }
1Injects the my-data-stream channel using the @Channel qualifier
2Indicates that the content is sent using Server Sent Events
3Indicates that the data contained within the server sent events is of type text/plain
4Returns the stream (Reactive Stream)

Configuring the Kafka connector

We need to configure the Kafka connector. This is done in the application.properties file.The keys are structured as follows:

mp.messaging.[outgoing|incoming].{channel-name}.property=value

The channel-name segment must match the value set in the @Incoming and @Outgoing annotation:

  • generated-price → sink in which we write the prices

  • prices → source in which we read the prices

  1. # Configure the Kafka sink (we write to it)
  2. mp.messaging.outgoing.generated-price.connector=smallrye-kafka
  3. mp.messaging.outgoing.generated-price.topic=prices
  4. mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer
  6. # Configure the Kafka source (we read from it)
  7. mp.messaging.incoming.prices.connector=smallrye-kafka
  8. mp.messaging.incoming.prices.value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer

More details about this configuration is available on the Producer configuration and Consumer configuration section from the Kafka documentation.

What about my-data-stream? This is an in-memory stream, not connected to a message broker.

The HTML page

Final touch, the HTML page reading the converted prices using SSE.

Create the src/main/resources/META-INF/resources/prices.html file, with the following content:

  1. <!DOCTYPE html>
  2. <html lang="en">
  3. <head>
  4.     <meta charset="UTF-8">
  5.     <title>Prices</title>
  7.     <link rel="stylesheet" type="text/css"
  8.           href="https://cdnjs.cloudflare.com/ajax/libs/patternfly/3.24.0/css/patternfly.min.css">
  9.     <link rel="stylesheet" type="text/css"
  10.           href="https://cdnjs.cloudflare.com/ajax/libs/patternfly/3.24.0/css/patternfly-additions.min.css">
  11. </head>
  12. <body>
  13. <div class="container">
  15.     <h2>Last price</h2>
  16.     <div class="row">
  17.     <p class="col-md-12">The last price is <strong><span id="content">N/A</span>&nbsp;&euro;</strong>.</p>
  18.     </div>
  19. </div>
  20. </body>
  21. <script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
  22. <script>
  23.     var source = new EventSource("/prices/stream");
  24.     source.onmessage = function (event) {
  25.         document.getElementById("content").innerHTML = event.data;
  26.     };
  27. </script>
  28. </html>

Nothing spectacular here. On each received price, it updates the page.

Get it running

If you followed the instructions, you should have Kafka running.Then, you just need to run the application using:

  1. ./mvnw compile quarkus:dev

Open http://localhost:8080/prices.html in your browser.

If you started the Kafka broker with docker compose, stop it using CTRL+C followed by docker-compose down.

Running Native

You can build the native executable with:

  1. ./mvnw package -Pnative

Going further

This guide has shown how you can interact with Kafka using Quarkus.It utilizes MicroProfile Reactive Messaging to build data streaming applications.

If you want to go further check the documentation of SmallRye Reactive Messaging, the implementation used in Quarkus.