Ingest Data into GCP Pub/Sub

EMQX supports seamless integration with Google Cloud Pub/SubIngest Data into GCP Pub/Sub - 图1 (opens new window) for real-time extraction, processing and analysis of MQTT data, and can also push and subscribe to various Google Cloud services such as Cloud Functions, App Engine, Cloud Run or Kubernetes Engine or Compute Engine.

EMQX GCP Pub/Sub integration allows you to send MQTT messages and events to GCP Pub/Sub, which can help you flexibly choose services on Google Cloud and build IoT applications more easily.

TIP

EMQX Enterprise Edition features. EMQX Enterprise Edition provides comprehensive coverage of key business scenarios, rich data integration, product-level reliability, and 24/7 global technical support. Experience the benefits of this enterprise-ready MQTT messaging platformIngest Data into GCP Pub/Sub - 图2 (opens new window) today.

Prerequisites

Feature List

Quick Start Tutorial

This section introduces how to configure the GCP Pub/Sub data bridge, including how to set up the GCP service, create data bridges and rules for forwarding data to GCP and test the data bridges and rules.

Create Service Account Key in GCP

You need to create a service account and a service account key to use the GCP PubSub service.

  1. Create a Service AccountIngest Data into GCP Pub/Sub - 图3 (opens new window) in your GCP account. Ensure that the Service Account has permissions to at least publish messages to the topic of interest.

  2. Click the email address for the service account you created. Click the Key tab. In the Add key drop-down list, select Create new key to create a Sercice Account key for that account and download it in JSON format.

    TIP

    Store the Service Account key securely for later use.

    service-account-key

Create and Manage Topics in GCP

Before configuring the GCP Pub/Sub Bridge on EMQX, you need to create a topic and be familiar with the basic management operation in GCP.

  1. In the Google Cloud console, go to the Pub/Sub ->Topics page. For detailed instructions, see Create and manage topicsIngest Data into GCP Pub/Sub - 图5 (opens new window).

    TIP

    The Service Account must have permission to publish that topic.

  2. In the Topic ID field, enter an ID for your topic. Click Create topic.

    create-topic-GCP-console

  3. Go to the Subscriptions page. Click the Topic ID in the list. Create a subscription to the topic.

    • Select Pull in Delivery type.
    • Select 7 Days for Message retention duration.

    For detailed instructions, see GCP Pub/Sub SubscriptionIngest Data into GCP Pub/Sub - 图7 (opens new window).

    add-subscription-to-topic

  4. Click Subscription ID -> Messages -> Pull can view the message sent to the topic.

    subscriptions-id subscriptions-id-pull

Create a GCP Pub/Sub Bridge

  1. Go to EMQX Dashboard, click Integration -> Data Bridge.

  2. Click Create on the top right corner of the page.

  3. In the Create Data Bridge page, click to select Google PubSub, and then click Next.

  4. Input a name for the data bridge. The name should be a combination of upper/lower case letters and numbers.

  5. In GCP PubSub Topic field, input the topic id you created in Create and Manage Topic in GCP.

  6. In the Payload Template field, leave it blank or define a template.

    • If left blank, it will encode all visible inputs from the MQTT message using JSON format, such as clientid, topic, payload, etc.
    • If use the defined template, placeholders of the form ${variable_name} will be filled with the corresponding value from the MQTT context. For example, ${topic} will be replaced with my/topic if such is the MQTT message topic.

  7. In the GCP Service Account Credentials field, upload the Service Account credentials in JSON format you exported in Create Service Account Key in GCP.

  8. Advanced settings (optional): Choose whether to use sync or async query mode as needed.

  9. Before clicking Create, you can click Test Connectivity to test that the bridge can connect to the Redis server.

  10. Click Create to finish the creation of the data bridge.

    A confirmation dialog will appear and ask if you like to create a rule using this data bridge, you can click Create Rule to continue creating rules to specify the data to be saved into GCP PubSub. You can also create rules by following the steps in Create Rules for GCP PubSub Data Bridge.

Create a Rule for GCP PubSub Data Bridge

You can continue to create rules to specify the data to be saved into GCP PubSub.

  1. Go to EMQX Dashboard, click Integration -> Rules.

  2. Click Create on the top right corner of the page.

  3. Input my_rule as the rule ID.

  4. Set the rules in the SQL Editor. Here we want to save the MQTT messages under topic /devices/+/events to GCP PubSub, we can use the SQL syntax below.

    Note: If you want to specify your own SQL syntax, make sure that the SELECT part includes all fields required by the payload template in the data bridge.

    1. SELECT
    2.   *
    3. FROM
    4.   "/devices/+/events"

  5. Click the Add Action button, select Forwarding with Data Bridge from the dropdown list, and then select the data bridge you just created under Data Bridge. Then click the Add button.

  6. Click Create at the page bottom to finish the creation.

Now a rule to forward data to GCP PubSub via a GCP PubSub bridge is created. You can click Integration -> Flows to view the topology. It can be seen that the messages under topic /devices/+/events are sent and saved to GCP PubSub after parsing by rule my_rule.

Test the Data Bridge and Rule

  1. Use MQTTX to send messages on the topic /devices/+/events.

    1. mqttx pub -i emqx_c -t /devices/+/events -m '{ "msg": "hello GCP PubSub" }'

  2. Check the running status of the data bridge, there should be one new incoming and one new outgoing message.

  3. Go to GCP Pub/Sub -> Subscriptions, click MESSAGES tab. You should see the message.