Azure Event Hubs

Detailed documentation on the Azure Event Hubs pubsub component

Component format

To setup Azure Event Hubs pubsub create a component of type pubsub.azure.eventhubs. See this guide on how to create and apply a pubsub configuration. Apart from the configuration metadata fields shown below, Azure Event Hubs also supports Azure Authentication mechanisms.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: eventhubs-pubsub
  5. spec:
  6. type: pubsub.azure.eventhubs
  7. version: v1
  8. metadata:
  9. - name: connectionString # Either connectionString or eventHubNamespace. Should not be used when
  10. # Azure Authentication mechanism is used.
  11. value: "Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}"
  12. - name: eventHubNamespace # Either connectionString or eventHubNamespace. Should be used when
  13. # Azure Authentication mechanism is used.
  14. value: "namespace"
  15. - name: enableEntityManagement
  16. value: "false"
  17. ## The following four properties are needed only if enableEntityManagement is set to true
  18. - name: resourceGroupName
  19. value: "test-rg"
  20. - name: subscriptionID
  21. value: "value of Azure subscription ID"
  22. - name: partitionCount
  23. value: "1"
  24. - name: messageRetentionInDays
  25. ## Subscriber attributes
  26. - name: storageAccountName
  27. value: "myeventhubstorage"
  28. - name: storageAccountKey
  29. value: "112233445566778899"
  30. - name: storageContainerName
  31. value: "myeventhubstoragecontainer"

Warning

The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described here.

Spec metadata fields

FieldRequiredDetailsExample
connectionStringYConnection-string for the Event Hub or the Event Hub namespace. Mutally exclusive with eventHubNamespace field. Not to be used when Azure Authentication is used“Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key};EntityPath={EventHub}” or “Endpoint=sb://{EventHubNamespace}.servicebus.windows.net/;SharedAccessKeyName={PolicyName};SharedAccessKey={Key}”
eventHubNamespaceNThe Event Hub Namespace name. Mutally exclusive with connectionString field. To be used when Azure Authentication is used“namespace”
storageAccountNameYStorage account name to use for the EventProcessorHost“myeventhubstorage”
storageAccountKeyYStorage account key to use for the EventProcessorHost. Can be secretKeyRef to use a secret reference. Omit if using Azure Authentication and AAD authentication to the storage account is preferred.“112233445566778899”
storageContainerNameYStorage container name for the storage account name.“myeventhubstoragecontainer”
enableEntityManagementNBoolean value to allow management of EventHub namespace. Default: false“true”, “false”
resourceGroupNameNName of the resource group the event hub namespace is a part of. Needed when entity management is enabled“test-rg”
subscriptionIDNAzure subscription ID value. Needed when entity management is enabled“azure subscription id”
partitionCountNNumber of partitions for the new event hub. Only used when entity management is enabled. Default: “1”“2”
messageRetentionInDaysNNumber of days to retain messages for in the newly created event hub. Used only when entity management is enabled. Default: “1”“90”

Azure Active Directory (AAD) authentication

The Azure Event Hubs pubsub component supports authentication using all Azure Active Directory mechanisms. For further information and the relevant component metadata fields to provide depending on the choice of AAD authentication mechanism, see the docs for authenticating to Azure.

Example Configuration

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Component
  3. metadata:
  4. name: eventhubs-pubsub
  5. spec:
  6. type: pubsub.azure.eventhubs
  7. version: v1
  8. metadata:
  9. # Azure Authentication Used
  10. - name: azureTenantId
  11. value: "***"
  12. - name: azureClientId
  13. value: "***"
  14. - name: azureClientSecret
  15. value: "***"
  16. - name: eventHubNamespace
  17. value: "namespace"
  18. - name: enableEntityManagement
  19. value: "false"
  20. ## The following four properties are needed only if enableEntityManagement is set to true
  21. - name: resourceGroupName
  22. value: "test-rg"
  23. - name: subscriptionID
  24. value: "value of Azure subscription ID"
  25. - name: partitionCount
  26. value: "1"
  27. - name: messageRetentionInDays
  28. ## Subscriber attributes
  29. - name: storageAccountName
  30. value: "myeventhubstorage"
  31. - name: storageAccountKey
  32. value: "112233445566778899"
  33. - name: storageContainerName
  34. value: "myeventhubstoragecontainer"

Create an Azure Event Hub

Follow the instructions here on setting up Azure Event Hubs. Since this implementation uses the Event Processor Host, you will also need an Azure Storage Account. Follow the instructions here to manage the storage account access keys.

See here on how to get the Event Hubs connection string. Note this is not the Event Hubs namespace.

Create consumer groups for each subscriber

For every Dapr app that wants to subscribe to events, create an Event Hubs consumer group with the name of the dapr id. For example, a Dapr app running on Kubernetes with dapr.io/app-id: "myapp" will need an Event Hubs consumer group named myapp.

Note: Dapr passes the name of the Consumer group to the EventHub and so this is not supplied in the metadata.

Entity Management

When entity management is enabled in configuration, as long as the application has the right role and permissions to manipulate the Event Hub namespace, creation of Event Hubs and consumer groups can be done on the fly.

The Evet Hub name is the topic field in the incoming request to publish or subscribe to, while the consumer group name is the name of the dapr app which subscribes to a given Event Hub. For example, a Dapr app running on Kubernetes with name dapr.io/app-id: "myapp" requires an Event Hubs consumer group named myapp.

Entity management is only possible when using Azure Authentication mechanisms and not via connectionString.

Note: Dapr passes the name of the Consumer group to the EventHub and this is not supplied in the metadata.

Subscribing to Azure IoT Hub Events

Azure IoT Hub provides an endpoint that is compatible with Event Hubs, so the Azure Event Hubs pubsub component can also be used to subscribe to Azure IoT Hub events.

The device-to-cloud events created by Azure IoT Hub devices will contain additional IoT Hub System Properties, and the Azure Event Hubs pubsub component for Dapr will return the following as part of the response metadata:

System Property NameDescription & Routing Query Keyword
iothub-connection-auth-generation-idThe connectionDeviceGenerationId of the device that sent the message. See IoT Hub device identity properties.
iothub-connection-auth-methodThe connectionAuthMethod used to authenticate the device that sent the message.
iothub-connection-device-idThe deviceId of the device that sent the message. See IoT Hub device identity properties.
iothub-connection-module-idThe moduleId of the device that sent the message. See IoT Hub device identity properties.
iothub-enqueuedtimeThe enqueuedTime in RFC3339 format that the device-to-cloud message was received by IoT Hub.
message-idThe user-settable AMQP messageId.

For example, the headers of a delivered HTTP subscription message would contain:

  1. {
  2. 'user-agent': 'fasthttp',
  3. 'host': '127.0.0.1:3000',
  4. 'content-type': 'application/json',
  5. 'content-length': '120',
  6. 'iothub-connection-device-id': 'my-test-device',
  7. 'iothub-connection-auth-generation-id': '637618061680407492',
  8. 'iothub-connection-auth-method': '{"scope":"module","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}',
  9. 'iothub-connection-module-id': 'my-test-module-a',
  10. 'iothub-enqueuedtime': '2021-07-13T22:08:09Z',
  11. 'message-id': 'my-custom-message-id',
  12. 'x-opt-sequence-number': '35',
  13. 'x-opt-enqueued-time': '2021-07-13T22:08:09Z',
  14. 'x-opt-offset': '21560',
  15. 'traceparent': '00-4655608164bc48b985b42d39865f3834-ed6cf3697c86e7bd-01'
  16. }

Last modified July 27, 2022: Remove namespace element from component examples (#2647) (ff9de5c8)