How-To: Publish a message and subscribe to a topic

Learn how to send messages to a topic with one service and subscribe to that topic in another service

Introduction

Pub/Sub is a common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers.

Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. Dapr provides different implementation of the underlying system, and allows operators to bring in their preferred infrastructure, for example Redis Streams, Kafka, etc.

Step 1: Setup the Pub/Sub component

The first step is to setup the Pub/Sub component:

Redis Streams is installed by default on a local machine when running dapr init.

Verify by opening your components file under %UserProfile%\.dapr\components\pubsub.yaml on Windows or ~/.dapr/components/pubsub.yaml on Linux/MacOS:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: pubsub
  5. spec:
  6. type: pubsub.redis
  7. version: v1
  8. metadata:
  9. - name: redisHost
  10. value: localhost:6379
  11. - name: redisPassword
  12. value: ""

You can override this file with another Redis instance or another pubsub component by creating a components directory containing the file and using the flag --components-path with the dapr run CLI command.

To deploy this into a Kubernetes cluster, fill in the metadata connection details of your desired pubsub component in the yaml below, save as pubsub.yaml, and run kubectl apply -f pubsub.yaml.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: pubsub
  5. namespace: default
  6. spec:
  7. type: pubsub.redis
  8. version: v1
  9. metadata:
  10. - name: redisHost
  11. value: localhost:6379
  12. - name: redisPassword
  13. value: ""

Step 2: Subscribe to topics

Dapr allows two methods by which you can subscribe to topics:

  • Declaratively, where subscriptions are are defined in an external file.
  • Programmatically, where subscriptions are defined in user code

Note

Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependancy from the user code and allows for the use of an existing application to subscribe to topics. The programmatic approach implements the subscription in user code.

Declarative subscriptions

You can subscribe to a topic using the following Custom Resources Definition (CRD). Create a file named subscription.yaml and paste the following:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Subscription
  3. metadata:
  4. name: myevent-subscription
  5. spec:
  6. topic: deathStarStatus
  7. route: /dsstatus
  8. pubsubname: pubsub
  9. scopes:
  10. - app1
  11. - app2

The example above shows an event subscription to topic deathStarStatus, for the pubsub component pubsub.

  • The route field tells Dapr to send all topic messages to the /dsstatus endpoint in the app.
  • The scopes field enables this subscription for apps with IDs app1 and app2.

Set the component with:

Place the CRD in your ./components directory. When Dapr starts up, it will load subscriptions along with components.

Note: By default, Dapr loads components from $HOME/.dapr/components on MacOS/Linux and %USERPROFILE%\.dapr\components on Windows.

You can also override the default directory by pointing the Dapr CLI to a components path:

  1. dapr run --app-id myapp --components-path ./myComponents -- python3 app1.py

Note: If you place the subscription in a custom components path, make sure the Pub/Sub component is present also.

In Kubernetes, save the CRD to a file and apply it to the cluster:

  1. kubectl apply -f subscription.yaml

Example

Create a file named app1.py and paste in the following:

  1. import flask
  2. from flask import request, jsonify
  3. from flask_cors import CORS
  4. import json
  5. import sys
  6. app = flask.Flask(__name__)
  7. CORS(app)
  8. @app.route('/dsstatus', methods=['POST'])
  9. def ds_subscriber():
  10. print(request.json, flush=True)
  11. return json.dumps({'success':True}), 200, {'ContentType':'application/json'}
  12. app.run()

After creating app1.py ensure flask and flask_cors are installed:

  1. pip install flask
  2. pip install flask_cors

Then run:

  1. dapr --app-id app1 --app-port 5000 run python app1.py

After setting up the subscription above, download this javascript (Node > 4.16) into a app2.js file:

  1. const express = require('express')
  2. const bodyParser = require('body-parser')
  3. const app = express()
  4. app.use(bodyParser.json({ type: 'application/*+json' }));
  5. const port = 3000
  6. app.post('/dsstatus', (req, res) => {
  7. console.log(req.body);
  8. res.sendStatus(200);
  9. });
  10. app.listen(port, () => console.log(`consumer app listening on port ${port}!`))

Run this app with:

  1. dapr --app-id app2 --app-port 3000 run node app2.js

Programmatic subscriptions

To subscribe to topics, start a web server in the programming language of your choice and listen on the following GET endpoint: /dapr/subscribe. The Dapr instance will call into your app at startup and expect a JSON response for the topic subscriptions with:

  • pubsubname: Which pub/sub component Dapr should use
  • topic: Which topic to subscribe to
  • route: Which endpoint for Dapr to call on when a message comes to that topic

Example

  1. import flask
  2. from flask import request, jsonify
  3. from flask_cors import CORS
  4. import json
  5. import sys
  6. app = flask.Flask(__name__)
  7. CORS(app)
  8. @app.route('/dapr/subscribe', methods=['GET'])
  9. def subscribe():
  10. subscriptions = [{'pubsubname': 'pubsub',
  11. 'topic': 'deathStarStatus',
  12. 'route': 'dsstatus'}]
  13. return jsonify(subscriptions)
  14. @app.route('/dsstatus', methods=['POST'])
  15. def ds_subscriber():
  16. print(request.json, flush=True)
  17. return json.dumps({'success':True}), 200, {'ContentType':'application/json'}
  18. app.run()

After creating app1.py ensure flask and flask_cors are installed:

  1. pip install flask
  2. pip install flask_cors

Then run:

  1. dapr --app-id app1 --app-port 5000 run python app1.py
  1. const express = require('express')
  2. const bodyParser = require('body-parser')
  3. const app = express()
  4. app.use(bodyParser.json({ type: 'application/*+json' }));
  5. const port = 3000
  6. app.get('/dapr/subscribe', (req, res) => {
  7. res.json([
  8. {
  9. pubsubname: "pubsub",
  10. topic: "deathStarStatus",
  11. route: "dsstatus"
  12. }
  13. ]);
  14. })
  15. app.post('/dsstatus', (req, res) => {
  16. console.log(req.body);
  17. res.sendStatus(200);
  18. });
  19. app.listen(port, () => console.log(`consumer app listening on port ${port}!`))

Run this app with:

  1. dapr --app-id app2 --app-port 3000 run node app2.js

The /dsstatus endpoint matches the route defined in the subscriptions and this is where Dapr will send all topic messages to.

Step 3: Publish a topic

To publish a message to a topic, invoke the following endpoint on a Dapr instance:

  1. dapr publish --pubsub pubsub --topic deathStarStatus --data '{"status": "completed"}'

Begin by ensuring a Dapr sidecar is running:

  1. dapr --app-id myapp --port 3500 run

Then publish a message to the deathStarStatus topic:

  1. curl -X POST http://localhost:3500/v1.0/publish/pubsub/deathStarStatus -H "Content-Type: application/json" -d '{"status": "completed"}'

Begin by ensuring a Dapr sidecar is running:

  1. dapr --app-id myapp --port 3500 run

Then publish a message to the deathStarStatus topic:

  1. Invoke-RestMethod -Method Post -ContentType 'application/json' -Body '{"status": "completed"}' -Uri 'http://localhost:3500/v1.0/publish/pubsub/deathStarStatus'

Dapr automatically wraps the user payload in a Cloud Events v1.0 compliant envelope, using Content-Type header value for datacontenttype attribute.

Step 4: ACK-ing a message

In order to tell Dapr that a message was processed successfully, return a 200 OK response. If Dapr receives any other return status code than 200, or if your app crashes, Dapr will attempt to redeliver the message following At-Least-Once semantics.

Example

  1. @app.route('/dsstatus', methods=['POST'])
  2. def ds_subscriber():
  3. print(request.json, flush=True)
  4. return json.dumps({'success':True}), 200, {'ContentType':'application/json'}
  1. app.post('/dsstatus', (req, res) => {
  2. res.sendStatus(200);
  3. });

Next steps

Last modified February 16, 2021: Merge pull request #1235 from dapr/update-v0.11 (b4e9fbb)