Pulsar SQl Deployment and Configuration

Below is a list configurations for the Presto Pulsar connector and instruction on how to deploy a cluster.

Presto Pulsar Connector Configurations

There are several configurations for the Presto Pulsar Connector. The properties file that contain these configurations can be found at ${project.root}/conf/presto/catalog/pulsar.properties.The configurations for the connector and its default values are discribed below.

  1. # name of the connector to be displayed in the catalog
  2. connector.name=pulsar
  4. # the url of Pulsar broker service
  5. pulsar.broker-service-url=http://localhost:8080
  7. # URI of Zookeeper cluster
  8. pulsar.zookeeper-uri=localhost:2181
  10. # minimum number of entries to read at a single time
  11. pulsar.entry-read-batch-size=100
  13. # default number of splits to use per query
  14. pulsar.target-num-splits=4

Query Pulsar from Existing Presto Cluster

If you already have an existing Presto cluster, you can copy Presto Pulsar connector plugin to your existing cluster. You can download the archived plugin package via:

  1. $ wget https://archive.apache.org/dist/pulsar/pulsar-2.4.0/apache-pulsar-2.4.0-bin.tar.gz

Deploying a new cluster

Please note that the Getting Started guide shows you how to easily setup a standalone single node enviroment to experiment with.

Pulsar SQL is powered by Presto thus many of the configurations for deployment is the same for the Pulsar SQL worker.

You can use the same CLI args as the Presto launcher:

  1. $ ./bin/pulsar sql-worker --help
  2. Usage: launcher [options] command
  4. Commands: run, start, stop, restart, kill, status
  6. Options:
  7.   -h, --help            show this help message and exit
  8.   -v, --verbose         Run verbosely
  9.   --etc-dir=DIR         Defaults to INSTALL_PATH/etc
  10.   --launcher-config=FILE
  11.                         Defaults to INSTALL_PATH/bin/launcher.properties
  12.   --node-config=FILE    Defaults to ETC_DIR/node.properties
  13.   --jvm-config=FILE     Defaults to ETC_DIR/jvm.config
  14.   --config=FILE         Defaults to ETC_DIR/config.properties
  15.   --log-levels-file=FILE
  16.                         Defaults to ETC_DIR/log.properties
  17.   --data-dir=DIR        Defaults to INSTALL_PATH
  18.   --pid-file=FILE       Defaults to DATA_DIR/var/run/launcher.pid
  19.   --launcher-log-file=FILE
  20.                         Defaults to DATA_DIR/var/log/launcher.log (only in
  21.                         daemon mode)
  22.   --server-log-file=FILE
  23.                         Defaults to DATA_DIR/var/log/server.log (only in
  24.                         daemon mode)
  25.   -D NAME=VALUE         Set a Java system property

There is a set of default configs for the cluster located in ${project.root}/conf/presto that will be used by default. You can change them to customize your deployment

You can also set the worker to read from a different configuration directory as well as set a different directory for writing its data:

  1. $ ./bin/pulsar sql-worker run --etc-dir /tmp/incubator-pulsar/conf/presto --data-dir /tmp/presto-1

You can also start the worker as daemon process:

  1. $ ./bin sql-worker start

Deploying to a 3 node cluster

For example, if I wanted to deploy a Pulsar SQL/Presto cluster on 3 nodes, you can do the following:

First, copy the Pulsar binary distribution to all three nodes.

The first node, will run the Presto coordinator. The mininal configuration in ${project.root}/conf/presto/config.properties can be the following

  1. coordinator=true
  2. node-scheduler.include-coordinator=true
  3. http-server.http.port=8080
  4. query.max-memory=50GB
  5. query.max-memory-per-node=1GB
  6. discovery-server.enabled=true
  7. discovery.uri=<coordinator-url>

Also, modify pulsar.broker-service-url and pulsar.zookeeper-uri configs in ${project.root}/conf/presto/catalog/pulsar.properties on those nodes accordingly

Afterwards, you can start the coordinator by just running

$ ./bin/pulsar sql-worker run

For the other two nodes that will only serve as worker nodes, the configurations can be the following:

  1. coordinator=false
  2. http-server.http.port=8080
  3. query.max-memory=50GB
  4. query.max-memory-per-node=1GB
  5. discovery.uri=<coordinator-url>

Also, modify pulsar.broker-service-url and pulsar.zookeeper-uri configs in ${project.root}/conf/presto/catalog/pulsar.properties accordingly

You can also start the worker by just running:

$ ./bin/pulsar sql-worker run

You can check the status of your cluster from the SQL CLI. To start the SQL CLI:

  1. $ ./bin/pulsar sql --server <coordinate_url>

You can then run the following command to check the status of your nodes:

  1. presto> SELECT * FROM system.runtime.nodes;
  2.  node_id |        http_uri         | node_version | coordinator | state  
  3. ---------+-------------------------+--------------+-------------+--------
  4.  1       | http://192.168.2.1:8081 | testversion  | true        | active 
  5.  3       | http://192.168.2.2:8081 | testversion  | false       | active 
  6.  2       | http://192.168.2.3:8081 | testversion  | false       | active

For more information about deployment in Presto, please reference:

Deploying Presto