Pulsar command-line tools

Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.

All Pulsar command-line tools can be run from the bin directory of your installed Pulsar package. The following tools are currently documented:

Getting help

You can get help for any CLI tool, command, or subcommand using the —help flag, or -h for short. Here's an example:

  1. $ bin/pulsar broker help

pulsar

The pulsar tool is used to start Pulsar components, such as bookies and ZooKeeper, in the foreground.

These processes can also be started in the background, using nohup, using the pulsar-daemon tool, which has the same command interface as pulsar.

Usage:

  1. $ pulsar command

Commands:

  • bookie
  • broker
  • compact-topic
  • discovery
  • configuration-store
  • initialize-cluster-metadata
  • proxy
  • standalone
  • websocket
  • zookeeper
  • zookeeper-shellExample:
  1. $ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

The table below lists the environment variables that you can use to configure the pulsar tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_BROKER_CONFConfiguration file for brokerconf/broker.conf
PULSAR_BOOKKEEPER_CONFdescription: Configuration file for bookieconf/bookkeeper.conf
PULSAR_ZK_CONFConfiguration file for zookeeperconf/zookeeper.conf
PULSAR_CONFIGURATION_STORE_CONFConfiguration file for the configuration storeconf/global_zookeeper.conf
PULSAR_DISCOVERY_CONFConfiguration file for discovery serviceconf/discovery.conf
PULSAR_WEBSOCKET_CONFConfiguration file for websocket proxyconf/websocket.conf
PULSAR_STANDALONE_CONFConfiguration file for standaloneconf/standalone.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the jvm
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath
PULSAR_PID_DIRFolder where the pulsar server PID file should be stored
PULSAR_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful

bookie

Starts up a bookie server

Usage:

  1. $ pulsar bookie options

Options

OptionDescriptionDefault
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recover service bookie serverfalse

Example

  1. $ PULSAR_BOOKKEEPER_CONF=/path/to/bookkeeper.conf pulsar bookie \
  2. -readOnly \
  3. -withAutoRecovery

broker

Starts up a Pulsar broker

Usage

  1. $ pulsar broker options

Options

OptionDescriptionDefault
-bc , —bookie-confConfiguration file for BookKeeper
-rb , —run-bookieRun a BookKeeper bookie on the same host as the Pulsar brokerfalse
-ra , —run-bookie-autorecoveryRun a BookKeeper autorecovery daemon on the same host as the Pulsar brokerfalse

Example

  1. $ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

compact-topic

Run compaction against a Pulsar topic (in a new process)

Usage

  1. $ pulsar compact-topic options

Options

FlagDescriptionDefault
-t , —topicThe Pulsar topic that you would like to compact

Example

  1. $ pulsar compact-topic --topic topic-to-compact

discovery

Run a discovery server

Usage

  1. $ pulsar discovery

Example

  1. $ PULSAR_DISCOVERY_CONF=/path/to/discovery.conf pulsar discovery

configuration-store

Starts up the Pulsar configuration store

Usage

  1. $ pulsar configuration-store

Example

  1. $ PULSAR_CONFIGURATION_STORE_CONF=/path/to/configuration_store.conf pulsar configuration-store

initialize-cluster-metadata

One-time cluster metadata initialization

Usage

  1. $ pulsar initialize-cluster-metadata options

Options

FlagDescriptionDefault
-ub , —broker-service-urlThe broker service URL for the new cluster
-tb , —broker-service-url-tlsThe broker service URL for the new cluster with TLS encryption
-c , —clusterCluster name
—configuration-storeThe configuration store quorum connection string
-uw , —web-service-urlThe web service URL for the new cluster
-tw , —web-service-url-tlsThe web service URL for the new cluster with TLS encryption
-zk , —zookeeperThe local ZooKeeper quorum connection string

proxy

Manages the Pulsar proxy

Usage

  1. $ pulsar proxy options

Options

FlagDescriptionDefault
—configuration-storeConfiguration store connection string
-zk , —zookeeper-serversLocal ZooKeeper connection string

Example

  1. $ PULSAR_PROXY_CONF=/path/to/proxy.conf pulsar proxy \
  2. --zookeeper-servers zk-0,zk-1,zk2 \
  3. --configuration-store zk-0,zk-1,zk-2

standalone

Run a broker service with local bookies and local ZooKeeper

Usage

  1. $ pulsar standalone options

Options

FlagDescriptionDefault
-a , —advertised-addressThe standalone broker advertised address
—bookkeeper-dirLocal bookies’ base data directorydata/standalone/bookeeper
—bookkeeper-portLocal bookies’ base port3181
—no-brokerOnly start ZooKeeper and BookKeeper services, not the brokerfalse
—num-bookiesThe number of local bookies1
—only-brokerOnly start the Pulsar broker service (not ZooKeeper or BookKeeper)
—wipe-dataClean up previous ZooKeeper/BookKeeper data
—zookeeper-dirLocal ZooKeeper’s data directorydata/standalone/zookeeper
—zookeeper-portLocal ZooKeeper’s port2181

Example

  1. $ PULSAR_STANDALONE_CONF=/path/to/standalone.conf pulsar standalone

websocket

Usage

  1. $ pulsar websocket

Example

  1. $ PULSAR_WEBSOCKET_CONF=/path/to/websocket.conf pulsar websocket

zookeeper

Starts up a ZooKeeper cluster

Usage

  1. $ pulsar zookeeper

Example

  1. $ PULSAR_ZK_CONF=/path/to/zookeeper.conf pulsar zookeeper

zookeeper-shell

Connects to a running ZooKeeper cluster using the ZooKeeper shell

Usage

  1. $ pulsar zookeeper-shell options

Options

FlagDescriptionDefault
-c, —confConfiguration file for ZooKeeper

pulsar-client

The pulsar-client tool

Usage

  1. $ pulsar-client command

Commands

  • produce
  • consumeOptions
FlagDescriptionDefault
—auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}"{"saslJaasClientSectionName":"PulsarClient", "serverType":"broker"}
—auth-pluginAuthentication plugin class nameorg.apache.pulsar.client.impl.auth.AuthenticationSasl
—urlBroker URL to which to connectpulsar://localhost:6650/

produce

Send a message or messages to a specific broker and topic

Usage

  1. $ pulsar-client produce topic options

Options

FlagDescriptionDefault
-f, —filesComma-separated file paths to send; either -m or -f must be specified[]
-m, —messagesComma-separated string of messages to send; either -m or -f must be specified[]
-n, —num-produceThe number of times to send the message(s); the count of messages/files * num-produce should be below 10001
-r, —rateRate (in messages per second) at which to produce; a value 0 means to produce messages as fast as possible0.0

consume

Consume messages from a specific broker and topic

Usage

  1. $ pulsar-client consume topic options

Options

FlagDescriptionDefault
—hexDisplay binary messages in hexadecimal format.false
-n, —num-messagesNumber of messages to consume, 0 means to consume forever.1
-r, —rateRate (in messages per second) at which to consume; a value 0 means to consume messages as fast as possible0.0
-s, —subscription-nameSubscription name
-t, —subscription-typeThe type of the subscription. Possible values: Exclusive, Shared, Failover, Key_Shared.Exclusive

pulsar-daemon

A wrapper around the pulsar tool that’s used to start and stop processes, such as ZooKeeper, bookies, and Pulsar brokers, in the background using nohup.

pulsar-daemon has a similar interface to the pulsar command but adds start and stop commands for various services. For a listing of those services, run pulsar-daemon to see the help output or see the documentation for the pulsar command.

Usage

  1. $ pulsar-daemon command

Commands

  • start
  • stop

start

Start a service in the background using nohup.

Usage

  1. $ pulsar-daemon start service

stop

Stop a service that’s already been started using start.

Usage

  1. $ pulsar-daemon stop service options

Options

FlagDescriptionDefault
-forceStop the service forcefully if not stopped by normal shutdown.false

pulsar-perf

A tool for performance testing a Pulsar broker.

Usage

  1. $ pulsar-perf command

Commands

  • consume
  • produce
  • read
  • websocket-producer
  • managed-ledger
  • monitor-brokers
  • simulation-client
  • simulation-controller
  • helpEnvironment variables

The table below lists the environment variables that you can use to configure the pulsar-perf tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_CLIENT_CONFConfiguration file for the clientconf/client.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the JVM
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath

consume

Run a consumer

Usage

  1. $ pulsar-perf consume options

Options

FlagDescriptionDefault
—auth_paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}.
—auth_pluginAuthentication plugin class name
—acks-delay-millisAcknowlegments grouping delay in millis100
-k, —encryption-key-nameThe private key name to decrypt payload
-v, —encryption-key-value-fileThe file which contains the private key to decrypt payload
-h, —helpHelp messagefalse
—conf-fileConfiguration file
-c, —max-connectionsMax number of TCP connections to a single broker100
-n, —num-consumersNumber of consumers (per topic)1
-t, —num-topicThe number of topics1
-r, —rateSimulate a slow message consumer (rate in msg/s)0
-q, —receiver-queue-sizeSize of the receiver queue1000
-u, —service-urlPulsar service URL
-i, —stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled0
-s, —subscriber-nameSubscriber name prefixsub
-st, —subscription-typeSubscriber name prefix. Possible values are Exclusive, Shared, Failover.Exclusive
—trust-cert-filePath for the trusted TLS certificate file

produce

Run a producer

Usage

  1. $ pulsar-perf produce options

Options

FlagDescriptionDefault
—auth_paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}.
—auth_pluginAuthentication plugin class name
-b, —batch-time-windowBatch messages in a window of the specified number of milliseconds1
-z, —compressionCompress messages’ payload. Possible values are NONE, LZ4, ZLIB, ZSTD or SNAPPY.
—conf-fileConfiguration file
-k, —encryption-key-nameThe public key name to encrypt payload
-v, —encryption-key-value-fileThe file which contains the public key to encrypt payload
-h, —helpHelp messagefalse
-c, —max-connectionsMax number of TCP connections to a single broker100
-o, —max-outstandingMax number of outstanding messages1000
-p, —max-outstanding-across-partitionsMax number of outstanding messages across partitions50000
-m, —num-messagesNumber of messages to publish in total. If set to 0, it will keep publishing.0
-n, —num-producersThe number of producers (per topic)1
-t, —num-topicThe number of topics1
-f, —payload-fileUse payload from an UTF-8 encoded text file and a payload will be randomly selected when publishing messages
-e, —payload-delimiterThe delimiter used to split lines when using payload from a file\n
-r, —ratePublish rate msg/s across topics100
-u, —service-urlPulsar service URL
-s, —sizeMessage size (in bytes)1024
-i, —stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
-time, —test-durationTest duration in secs. If set to 0, it will keep publishing.0
—trust-cert-filePath for the trusted TLS certificate file
—warmup-timeWarm-up time in seconds1

read

Run a topic reader

Usage

  1. $ pulsar-perf read options

Options

FlagDescriptionDefault
—auth_paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}.
—auth_pluginAuthentication plugin class name
—conf-fileConfiguration file
-h, —helpHelp messagefalse
-c, —max-connectionsMax number of TCP connections to a single broker100
-t, —num-topicThe number of topics1
-r, —rateSimulate a slow message reader (rate in msg/s)0
-q, —receiver-queue-sizeSize of the receiver queue1000
-u, —service-urlPulsar service URL
-m, —start-message-idStart message id. This can be either 'earliest', 'latest' or a specific message id by using 'lid:eid'earliest
-i, —stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
—trust-cert-filePath for the trusted TLS certificate file
—use-tlsUse TLS encryption on the connectionfalse

websocket-producer

Run a websocket producer

Usage

  1. $ pulsar-perf websocket-producer options

Options

FlagDescriptionDefault
—auth_paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}.
—auth_pluginAuthentication plugin class name
—conf-fileConfiguration file
-h, —helpHelp messagefalse
-m, —num-messagesNumber of messages to publish in total. If 0, it will keep publishing0
-t, —num-topicThe number of topics1
-f, —payload-fileUse payload from a file instead of empty buffer
-u, —proxy-urlPulsar Proxy URL, e.g., "ws://localhost:8080/"
-r, —ratePublish rate msg/s across topics100
-s, —sizeMessage size in byte1024
-time, —test-durationTest duration in secs. If 0, it will keep publishing0

managed-ledger

Write directly on managed-ledgers

Usage

  1. $ pulsar-perf managed-ledger options

Options

FlagDescriptionDefault
-a, —ack-quorumLedger ack quorum1
-dt, —digest-typeBookKeeper digest type. Possible Values: [CRC32, MAC, CRC32C, DUMMY]CRC32C
-e, —ensemble-sizeLedger ensemble size1
-h, —helpHelp messagefalse
-c, —max-connectionsMax number of TCP connections to a single bookie1
-o, —max-outstandingMax number of outstanding requests1000
-m, —num-messagesNumber of messages to publish in total. If 0, it will keep publishing0
-t, —num-topicNumber of managed ledgers1
-r, —rateWrite rate msg/s across managed ledgers100
-s, —sizeMessage size in byte1024
-time, —test-durationTest duration in secs. If 0, it will keep publishing0
—threadsNumber of threads writing1
-w, —write-quorumLedger write quorum1
-zk, —zookeeperServersZooKeeper connection string

monitor-brokers

Continuously receive broker data and/or load reports

Usage

  1. $ pulsar-perf monitor-brokers options

Options

FlagDescriptionDefault
—connect-stringA connection string for one or more ZooKeeper servers
-h, —helpHelp messagefalse

simulation-client

Run a simulation server acting as a Pulsar client. Uses the client configuration specified in conf/client.conf.

Usage

  1. $ pulsar-perf simulation-client options

Options

FlagDescriptionDefault
—portPort to listen on for controller0
—service-urlPulsar Service URL
-h, —helpHelp messagefalse

simulation-controller

Run a simulation controller to give commands to servers

Usage

  1. $ pulsar-perf simulation-controller options

Options

FlagDescriptionDefault
—client-portThe port that the clients are listening on0
—clientsComma-separated list of client hostnames
—clusterThe cluster to test on
-h, —helpHelp messagefalse

help

This help message

Usage

  1. $ pulsar-perf help

bookkeeper

A tool for managing BookKeeper.

Usage

  1. $ bookkeeper command

Commands

  • auto-recovery
  • bookie
  • localbookie
  • upgrade
  • shellEnvironment variables

The table below lists the environment variables that you can use to configure the bookkeeper tool.

VariableDescriptionDefault
BOOKIE_LOG_CONFLog4j configuration fileconf/log4j2.yaml
BOOKIE_CONFBookKeeper configuration fileconf/bk_server.conf
BOOKIE_EXTRA_OPTSExtra options to be passed to the JVM
BOOKIE_EXTRA_CLASSPATHExtra paths for BookKeeper's classpath
ENTRY_FORMATTER_CLASSThe Java class used to format entries
BOOKIE_PID_DIRFolder where the BookKeeper server PID file should be stored
BOOKIE_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful

auto-recovery

Runs an auto-recovery service daemon

Usage

  1. $ bookkeeper auto-recovery options

Options

FlagDescriptionDefault
-c, —confConfiguration for the auto-recovery daemon

bookie

Starts up a BookKeeper server (aka bookie)

Usage

  1. $ bookkeeper bookie options

Options

FlagDescriptionDefault
-c, —confConfiguration for the auto-recovery daemon
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recovery service bookie serverfalse

localbookie

Runs a test ensemble of N bookies locally

Usage

  1. $ bookkeeper localbookie N

upgrade

Upgrade the bookie’s filesystem

Usage

  1. $ bookkeeper upgrade options

Options

FlagDescriptionDefault
-c, —confConfiguration for the auto-recovery daemon
-u, —upgradeUpgrade the bookie’s directories

shell

Run shell for admin commands. To see a full listing of those commands, run bookkeeper shell without an argument.

Usage

  1. $ bookkeeper shell

Example

  1. $ bookkeeper shell bookiesanity