提示

在 Pulsar 部署中启用所有内置的 Pulsar IO connector，可以使用 apachepulsar/pulsar-all 镜像而不是 apachepulsar/pulsar 镜像。 apachepulsar/pulsar-all 镜像已经捆绑了所有内置 connector。

DC/OS (the DataCenter Operating System) is a distributed operating system used for deploying and managing applications and systems on Apache Mesos. DC/OS is an open-source tool created and maintained by Mesosphere.

Apache Pulsar is available as a Marathon Application Group, which runs multiple applications as manageable sets.

先决条件

In order to run Pulsar on DC/OS, you will need the following:

• DC/OS version 1.9 or higher

• A DC/OS cluster with at least three agent nodes

• The DC/OS CLI tool installed

• The PulsarGroups.json configuration file from the Pulsar GitHub repo.

  $ curl -O https://raw.githubusercontent.com/apache/pulsar/master/deployment/dcos/PulsarGroups.json

Each node in the DC/OS-managed Mesos cluster must have at least:

• 4 CPU
• 4 GB of memory
• 60 GB of total persistent disk

Alternatively, you can change the configuration in PulsarGroups.json according to match your DC/OS cluster’s resources.

Deploy Pulsar using the DC/OS command interface

You can deploy Pulsar on DC/OS using this command:

$ dcos marathon group add PulsarGroups.json

This command will deploy Docker container instances in three groups, which together comprise a Pulsar cluster:

• 3 bookies (1 bookie on each agent node and 1 bookie recovery instance)
• 3 Pulsar brokers (1 broker on each node and 1 admin instance)
• 1 Prometheus instance and 1 Grafana instance

When running DC/OS, a ZooKeeper cluster is already running at master.mesos:2181, thus there’s no need to install or start up ZooKeeper separately.

After executing the dcos command above, click on the Services tab in the DC/OS GUI interface, which you can access at http://m1.dcos in this example. You should see several applications in the process of deploying.

The BookKeeper group

To monitor the status of the BookKeeper cluster deployment, click on the bookkeeper group in the parent pulsar group.

At this point, 3 bookies should be shown as green, which means that they have been deployed successfully and are now running.

You can also click into each bookie instance to get more detailed info, such as the bookie running log.

To display information about the BookKeeper in ZooKeeper, you can visit http://m1.dcos/exhibitor. 在这个示例中，available 目录下有三个bookies

The Pulsar broker Group

Similar to the BookKeeper group above, click into the brokers to check the status of the Pulsar brokers.

你也可以点击每个broker实例来获取更多的详细清晰，比如broker运行日志。

Broker cluster information in Zookeeper is also available through the web UI. 在这个示例中，您可以看到 loadbalance 和 managed-ledgers 目录已被创建。

Monitor Group

The monitory group consists of Prometheus and Grafana.

Prometheus

Click into the instance of prom to get the endpoint of Prometheus, which is 192.168.65.121:9090 in this example.

如果您点击此端点，您会看到Prometheus 仪表板。 链接 http://192.168.65.121:9090/target 会显示所有的bookies和brokers实例。

Grafana

Click into grafana to get the endpoint for Grafana, which is 192.168.65.121:3000 in this example.

If you click that endpoint, you can access the Grafana dashbaord.

Run a simple Pulsar consumer and producer on DC/OS

既然我们已经完全部署好了 Pulsar 集群，现在就可以运行一个简单的消费者和生产者，在DC/OS上我们就能够看到显示出了 Pulsar。

Download and prepare the Pulsar Java tutorial

你可以通过 Pulsar Java tutorial 这个链接克隆pulsar的java版本教程 This repo contains a simple Pulsar consumer and producer (more info can be found in the repo’s README file).

$ git clone https://github.com/streamlio/pulsar-java-tutorial

Change the SERVICE_URL from pulsar://localhost:6650 to pulsar://a1.dcos:6650 in both ConsumerTutorial.java and ProducerTutorial.java. The pulsar://a1.dcos:6650 endpoint is for the broker service. Endpoint details for each broker instance can be fetched from the DC/OS GUI. a1.dcos is a DC/OS client agent, which runs a broker. This can also be replaced by the client agent IP address.

Now, change the message number from 10 to 10000000 in main method of ProducerTutorial.java so that it will produce more messages.

Now compile the project code using command:

$ mvn clean package

Run the consumer and producer

Execute this command to run the consumer:

$ mvn exec:java -Dexec.mainClass="tutorial.ConsumerTutorial"

Execute this command to run the producer:

$ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"

You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI.

View Grafana metric output

While the producer and consumer are running, you can access running metrics information from Grafana.

Uninstall Pulsar

You can shut down and uninstall the pulsar application from DC/OS at any time in two ways:

1. Using the DC/OS GUI, you can choose Delete at the right end of Pulsar group.

   

2. You can use the following command:

   $ dcos marathon group remove /pulsar