Pulsar SQL configuration and deployment

You can configure the Pulsar Trino plugin and deploy a cluster with the following instruction.

Configure Pulsar Trino plugin

You can configure the Pulsar Trino plugin in the ${project.root}/trino/conf/catalog/pulsar.properties properties file. The configuration for the connector and the default values are as follows.

  1. # name of the connector to be displayed in the catalog
  2. connector.name=pulsar
  4. # the URL of Pulsar broker service
  5. pulsar.web-service-url=http://localhost:8080
  7. # the URL of Pulsar broker binary service
  8. pulsar.broker-binary-service-url=pulsar://localhost:6650
  10. # the URL of Zookeeper cluster
  11. pulsar.zookeeper-uri=localhost:2181
  13. # minimum number of entries to read at a single time
  14. pulsar.entry-read-batch-size=100
  16. # default number of splits to use per query
  17. pulsar.target-num-splits=4
  19. # max size of one batch message (default value is 5MB)
  20. pulsar.max-message-size=5242880
  22. # number of split used when querying data from Pulsar
  23. pulsar.target-num-splits=2
  25. # size of queue to buffer entry read from Pulsar
  26. pulsar.max-split-entry-queue-size=1000
  28. # size of queue to buffer message extract from entries
  29. pulsar.max-split-message-queue-size=10000
  31. # status provider to record connector metrics
  32. pulsar.stats-provider=org.apache.bookkeeper.stats.NullStatsProvider
  34. # config in map format for stats provider e.g. {"key1":"val1","key2":"val2"}
  35. pulsar.stats-provider-configs={}
  37. # whether to rewrite Pulsar's default topic delimiter '/'
  38. pulsar.namespace-delimiter-rewrite-enable=false
  40. # delimiter used to rewrite Pulsar's default delimiter '/', use if default is causing incompatibility with other system like Superset
  41. pulsar.rewrite-namespace-delimiter="/"
  43. # maximum number of thread pool size for ledger offloader.
  44. pulsar.managed-ledger-offload-max-threads=2
  46. # driver used to offload or read cold data to or from long-term storage
  47. pulsar.managed-ledger-offload-driver=null
  49. # directory to load offloaders nar file.
  50. pulsar.offloaders-directory="./offloaders"
  52. # properties and configurations related to specific offloader implementation as map e.g. {"key1":"val1","key2":"val2"}
  53. pulsar.offloader-properties={}
  55. # authentication plugin used to authenticate to Pulsar cluster
  56. pulsar.auth-plugin=null
  58. # authentication parameter used to authenticate to the Pulsar cluster as a string e.g. "key1:val1,key2:val2".
  59. pulsar.auth-params=null
  61. # whether the Pulsar client accept an untrusted TLS certificate from broker
  62. pulsar.tls-allow-insecure-connection=null
  64. # whether to allow hostname verification when a client connects to broker over TLS.
  65. pulsar.tls-hostname-verification-enable=null
  67. # path for the trusted TLS certificate file of Pulsar broker
  68. pulsar.tls-trust-cert-file-path=null
  70. ## whether to enable Pulsar authorization
  71. pulsar.authorization-enabled=false
  73. # set the threshold for BookKeeper request throttle, default is disabled
  74. pulsar.bookkeeper-throttle-value=0
  76. # set the number of IO thread
  77. pulsar.bookkeeper-num-io-threads=2 * Runtime.getRuntime().availableProcessors()
  79. # set the number of worker thread
  80. pulsar.bookkeeper-num-worker-threads=Runtime.getRuntime().availableProcessors()
  82. # whether to use BookKeeper V2 wire protocol
  83. pulsar.bookkeeper-use-v2-protocol=true
  85. # interval to check the need for sending an explicit LAC, default is disabled
  86. pulsar.bookkeeper-explicit-interval=0
  88. # size for managed ledger entry cache (in MB).
  89. pulsar.managed-ledger-cache-size-MB=0
  91. # number of threads to be used for managed ledger tasks dispatching
  92. pulsar.managed-ledger-num-worker-threads=Runtime.getRuntime().availableProcessors()
  94. # number of threads to be used for managed ledger scheduled tasks
  95. pulsar.managed-ledger-num-scheduler-threads=Runtime.getRuntime().availableProcessors()
  97. # directory used to store extraction NAR file
  98. pulsar.nar-extraction-directory=System.getProperty("java.io.tmpdir")

Enable authentication and authorization between Pulsar and Pulsar SQL

By default, the authentication and authorization between Pulsar and Pulsar SQL are disabled.

To enable it, set the following configurations in the ${project.root}/trino/conf/catalog/pulsar.properties properties file:

  1. pulsar.authorization-enabled=true
  2. pulsar.broker-binary-service-url=pulsar://localhost:6650

Connect Trino to Pulsar with multiple hosts

You can connect Trino to a Pulsar cluster with multiple hosts.

  • To configure multiple hosts for brokers, add multiple URLs to pulsar.web-service-url.
  • To configure multiple hosts for ZooKeeper, add multiple URIs to pulsar.zookeeper-uri.

The following is an example.

  1. pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
  2. pulsar.zookeeper-uri=localhost1,localhost2:2181

Get the last message in a topic

Configuration and deployment - 图1note

By default, Pulsar SQL does not get the last message in a topic. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger.

If you want to get the last message in a topic, set the following configurations:

  1. For the broker configuration, set bookkeeperExplicitLacIntervalInMills > 0 in broker.conf or standalone.conf.

  2. For the Trino configuration, set pulsar.bookkeeper-explicit-interval > 0 and pulsar.bookkeeper-use-v2-protocol=false.

However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.

Query data from existing Trino clusters

If you already have a Trino cluster compatible to version 363, you can copy the Pulsar Trino plugin to your existing cluster. Download the archived plugin package with the following command.

  1. wget https://archive.apache.org/dist/pulsar/pulsar-3.1.1/apache-pulsar-3.1.1-bin.tar.gz

Deploy a new cluster

Since Pulsar SQL is powered by Trino, the configuration for deployment is the same for the Pulsar SQL worker.

Configuration and deployment - 图2note

For how to set up a standalone single node environment, refer to Query data.

You can use the same CLI args as the Trino launcher.

The default configuration for the cluster is located in ${project.root}/trino/conf. You can customize your deployment by modifying the default configuration.

You can set the worker to read from a different configuration directory, or set a different directory to write data.

  1. ./bin/pulsar sql-worker run --etc-dir /tmp/pulsar/trino/conf --data-dir /tmp/trino-1

You can start the worker as daemon process.

  1. ./bin/pulsar sql-worker start

Deploy a cluster on multiple nodes

You can deploy a Pulsar SQL cluster or Trino cluster on multiple nodes. The following example shows how to deploy a cluster on three-node cluster.

  1. Copy the Pulsar binary distribution to three nodes.

The first node runs as Trino coordinator. The minimal configuration required in the ${project.root}/trino/conf/config.properties file is as follows.

  1. coordinator=true
  2. node-scheduler.include-coordinator=true
  3. http-server.http.port=8080
  4. query.max-memory=50GB
  5. query.max-memory-per-node=1GB
  6. discovery-server.enabled=true
  7. discovery.uri=<coordinator-url>

The other two nodes serve as worker nodes, you can use the following configuration for worker nodes.

  1. coordinator=false
  2. http-server.http.port=8080
  3. query.max-memory=50GB
  4. query.max-memory-per-node=1GB
  5. discovery.uri=<coordinator-url>

  1. Modify pulsar.web-service-url and pulsar.zookeeper-uri configuration in the ${project.root}/trino/conf/catalog/pulsar.properties file accordingly for the three nodes.

  2. Start the coordinator node:

  1. ./bin/pulsar sql-worker run
  1. Start worker nodes:
  1. ./bin/pulsar sql-worker run
  1. Start the SQL CLI and check the status of your cluster:
  1. ./bin/pulsar sql --server <coordinate_url>
  1. Check the status of your nodes:
  1. trino> SELECT * FROM system.runtime.nodes;
  2.  node_id |        http_uri         | node_version | coordinator | state
  3. ---------+-------------------------+--------------+-------------+--------
  4.  1       | http://192.168.2.1:8081 | testversion  | true        | active
  5.  3       | http://192.168.2.2:8081 | testversion  | false       | active
  6.  2       | http://192.168.2.3:8081 | testversion  | false       | active

For more information about the deployment in Trino, refer to Trino deployment.

Configuration and deployment - 图3note

The broker does not advance LAC, so when Pulsar SQL bypasses brokers to query data, it can only read entries up to the LAC that all the bookies learned. You can enable periodically write LAC on the broker by setting “bookkeeperExplicitLacIntervalInMills” in the broker.conf file.