Pulsar metrics

Pulsar exposes the following metrics in Prometheus format. You can monitor your clusters with those metrics.

The following types of metrics are available:

  • Counter: a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster.
  • Gauge: a metric that represents a single numerical value that can arbitrarily go up and down.
  • Histogram: a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. The _bucket suffix is the number of observations within a histogram bucket, configured with parameter {le="<upper inclusive bound>"}. The _count suffix is the number of observations, shown as a time series and behaves like a counter. The _sum suffix is the sum of observed values, also shown as a time series and behaves like a counter. These suffixes are together denoted by _* in this doc.
  • Summary: similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

ZooKeeper

The ZooKeeper metrics are exposed under “/metrics” at port 8000. You can use a different port by configuring the metricsProvider.httpPort in conf/zookeeper.conf.

ZooKeeper provides a New Metrics System since 3.6.0. For more detailed metrics, refer to the ZooKeeper Monitor Guide.

BookKeeper

The BookKeeper metrics are exposed under “/metrics” at port 8000. You can change the port by updating prometheusStatsHttpPort in the bookkeeper.conf configuration file.

Server metrics

NameTypeDescription
bookie_SERVER_STATUSGaugeThe server status for bookie server.
  • 1: the bookie is running in writable mode.
  • 0: the bookie is running in read-only mode.
bookkeeper_server_ADD_ENTRY_countCounterThe total number of ADD_ENTRY requests received at the bookie. The success label is used to distinguish between successes and failures.
bookkeeper_server_READ_ENTRY_countCounterThe total number of READ_ENTRY requests received at the bookie. The success label is used to distinguish between successes and failures.
bookie_WRITE_BYTESCounterThe total number of bytes written to the bookie.
bookie_READ_BYTESCounterThe total number of bytes read from the bookie.
bookkeeper_server_ADD_ENTRY_REQUESTSummaryThe summary of request latency of ADD_ENTRY requests at the bookie. The success label is used to distinguish between successes and failures.
bookkeeper_server_READ_ENTRY_REQUESTSummaryThe summary of request latency of READ_ENTRY requests at the bookie. The success label is used to distinguish between successes and failures.
bookkeeperserver_BookieReadThreadPool_queue{thread_id}GaugeThe number of requests to be processed in a read thread queue.
bookkeeper_server_BookieReadThreadPool_task_queuedSummaryThe waiting time of a task to be processed in a read thread queue.
bookkeeper_server_BookieReadThreadPool_task_executionSummaryThe execution time of a task in a read thread queue.

Journal metrics

NameTypeDescription
bookie_journal_JOURNAL_SYNC_countCounterThe total number of journal fsync operations happening at the bookie. The success label is used to distinguish between successes and failures.
bookie_journal_JOURNAL_QUEUE_SIZEGaugeThe total number of requests pending in the journal queue.
bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZEGaugeThe total number of force write (fsync) requests pending in the force-write queue.
bookie_journal_JOURNAL_CB_QUEUE_SIZEGaugeThe total number of callbacks pending in the callback queue.
bookie_journal_JOURNAL_ADD_ENTRYSummaryThe summary of request latency of adding entries to the journal.
bookie_journal_JOURNAL_SYNCSummaryThe summary of fsync latency of syncing data to the journal disk.
bookie_journal_JOURNAL_CREATION_LATENCYSummaryThe latency created by a journal log file.

Storage metrics

NameTypeDescription
bookie_ledgers_countGaugeThe total number of ledgers stored in the bookie.
bookie_entries_countGaugeThe total number of entries stored in the bookie.
bookie_write_cache_sizeGaugeThe bookie write cache size (in bytes).
bookie_read_cache_sizeGaugeThe bookie read cache size (in bytes).
bookie_DELETED_LEDGER_COUNTCounterThe total number of ledgers deleted since the bookie has started.
bookie_ledger_writable_dirsGaugeThe number of writable directories in the bookie.
bookie_flushGaugeThe table flush latency of bookie memory.
bookie_throttled_write_requestsCounterThe number of write requests to be throttled.

Replication metrics

NameTypeDescription
auditor_NUM_UNDER_REPLICATED_LEDGERSSummaryThe distribution of num under_replicated ledgers on each auditor run.
auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZESummaryThe distribution of under_replicated ledgers total size on each auditor run.
auditor_URL_PUBLISH_TIME_FOR_LOST_BOOKIESummaryThe latency distribution of publishing under replicated ledgers for lost bookies.
auditor_BOOKIE_TO_LEDGERS_MAP_CREATION_TIMESummaryThe latency distribution of creating bookies-to-ledgers map.
auditor_CHECK_ALL_LEDGERS_TIMESummaryThe latency distribution of checking all ledgers.
auditor_PLACEMENT_POLICY_CHECK_TIMESummaryThe latency distribution of placementPolicy check.
auditor_REPLICAS_CHECK_TIMESummaryThe latency distribution of replicas check.
auditor_AUDIT_BOOKIES_TIMESummaryThe latency distribution of auditing all the bookies.
auditor_NUM_LEDGERS_CHECKEDCounterThe number of ledgers checked by the auditor.
auditor_NUM_FRAGMENTS_PER_LEDGERSummaryThe distribution of number of fragments per ledger.
auditor_NUM_BOOKIES_PER_LEDGERSummaryThe distribution of number of bookies per ledger.
auditor_NUM_BOOKIE_AUDITS_DELAYEDCounterThe number of bookie-audits delayed.
auditor_NUM_DELAYED_BOOKIE_AUDITS_DELAYES_CANCELLEDCounterThe number of delayed-bookie-audits cancelled.
auditor_NUM_LEDGERS_NOT_ADHERING_TO_PLACEMENT_POLICYGaugeThe number of ledgers not adhering to placement policy found in placement policy check.
auditor_NUM_LEDGERS_SOFTLY_ADHERING_TO_PLACEMENT_POLICYGaugeThe number of ledgers softly adhering to placement policy found in placement policy check.
auditor_NUM_UNDERREPLICATED_LEDGERS_ELAPSED_RECOVERY_GRACE_PERIODGaugeThe number of underreplicated ledgers elapsed recovery grace period.
auditor_NUM_LEDGERS_HAVING_NO_REPLICA_OF_AN_ENTRYGaugeThe number of ledgers having an entry with all the replicas missing.
auditor_NUM_LEDGERS_HAVING_LESS_THAN_AQ_REPLICAS_OF_AN_ENTRYGaugeThe number of ledgers having an entry with less than AQ number of replicas, this doesn’t include ledgers counted towards numLedgersHavingNoReplicaOfAnEntry.
auditor_NUM_LEDGERS_HAVING_LESS_THAN_WQ_REPLICAS_OF_AN_ENTRYGaugeThe number of ledgers having an entry with less than WQ number of replicas, this doesn’t include ledgers counted towards numLedgersHavingLessThanAQReplicasOfAnEntry.
election_attemptsCounterThe number of auditor election attempts.
replication_worker_NUM_BYTES_READSummaryThe distribution size of entries read by the replicator.
replication_worker_NUM_ENTRIES_READCounterNumber of entries read by the replicator.
replication_worker_NUM_ENTRIES_WRITTENCounterNumber of entries written by the replicator.
replication_worker_NUM_BYTES_WRITTENSummaryThe distribution size of entries written by the replicator.
replication_worker_READ_DATA_LATENCYSummaryThe distribution of latency of read entries by the replicator.
replication_worker_WRITE_DATA_LATENCYSummaryThe distribution of latency of write entries by the replicator.
replication_worker_exceptionsSummaryReplication related exceptions.
replication_worker_rereplicateSummaryOperation stats of re-replicating ledgers.
replication_worker_NUM_FULL_OR_PARTIAL_LEDGERS_REPLICATEDCounterThe number of ledgers re-replicated.
replication_worker_NUM_DEFER_LEDGER_LOCK_RELEASE_OF_FAILED_LEDGERCounterThe number of defer-ledger-lock-releases of failed ledgers.
replication_worker_NUM_ENTRIES_UNABLE_TO_READ_FOR_REPLICATIONCounterThe number of entries ReplicationWorker unable to read.

Broker

The broker metrics are exposed under “/metrics/“ at port 8080. You can change the port by updating webServicePort to a different port in the broker.conf configuration file.

All the metrics exposed by a broker are labeled with cluster=${pulsar_cluster}. The name of Pulsar cluster is the value of ${pulsar_cluster}, which you have configured in the broker.conf file.

Broker metrics

All the broker metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
pulsar_ml_cache_evictionsGaugeThe number of cache evictions during the last minute.
pulsar_ml_cache_inserted_entries_totalCounterThe number of entries inserted into the entry cache.
pulsar_ml_cache_evicted_entries_totalCounterThe number of entries evicted from the entry cache.
pulsar_ml_cache_entriesGaugeThe number of entries in the entry cache.
pulsar_ml_cache_hits_rateGaugeThe number of cache hits per second on the broker side.
pulsar_ml_cache_hits_throughputGaugeThe amount of data (byte per second) retrieved from the cache on the broker side.
pulsar_ml_cache_misses_rateGaugeThe number of cache missed per second on the broker side.
pulsar_ml_cache_misses_throughputGaugeThe amount of data (byte per second) that cannot be retrieved from the cache on the broker side.
pulsar_ml_cache_pool_active_allocationsGaugeThe number of currently active allocations in direct arena.
pulsar_ml_cache_pool_active_allocations_hugeGaugeThe number of currently active huge allocation in direct arena.
pulsar_ml_cache_pool_active_allocations_normalGaugeThe number of currently active normal allocations in direct arena.
pulsar_ml_cache_pool_active_allocations_smallGaugeThe number of currently active small allocations in direct arena.
pulsar_ml_cache_pool_allocatedGaugeThe total allocated memory of chunk lists in direct arena.
pulsar_ml_cache_pool_usedGaugeThe total used memory of chunk lists in direct arena.
pulsar_ml_cache_used_sizeGaugeThe size used to store the payloads of entries (in bytes).
pulsar_ml_countGaugeThe number of currently opened managed ledgers.
topic_load_times (deprecated)SummaryThe topic load latency calculated in milliseconds.
pulsar_topic_load_timesSummaryThe topic load latency calculated in milliseconds.
pulsar_active_connectionsGaugeThe number of active connections.
pulsar_connection_created_total_countGaugeThe total number of connections.
pulsar_connection_create_success_countGaugeThe number of successfully created connections.
pulsar_connection_create_fail_countGaugeThe number of failed connections.
pulsar_connection_closed_total_countGaugeThe total number of closed connections.
pulsar_broker_throttled_connectionsGaugeThe number of throttled connections.
pulsar_broker_throttled_connections_global_limitGaugeThe number of throttled connections due to per-connection limit.
pulsar_broker_topics_countGaugeThe number of Pulsar topics in this broker.
pulsar_broker_subscriptions_countGaugeThe number of Pulsar subscriptions in this broker.
pulsar_broker_producers_countGaugeThe number of active producers connected to this broker.
pulsar_broker_consumers_countGaugeThe number of active consumers connected to this broker.
pulsar_broker_rate_inGaugeThe total message rate coming into this broker (message per second).
pulsar_broker_rate_outGaugeThe total message rate going out from this broker (message per second).
pulsar_broker_throughput_inGaugeThe total throughput coming into this broker (byte per second).
pulsar_broker_throughput_outGaugeThe total throughput going out from this broker (byte per second).
pulsar_broker_storage_sizeGaugeThe total storage size of all topics in this broker (in bytes).
pulsar_broker_storage_logical_sizeGaugeThe storage size of all topics in this broker without replicas (in bytes).
pulsar_broker_storage_write_rateGaugeThe total message batches (entries) written to the storage for this broker (message batch per second).
pulsar_broker_storage_read_rateGaugeThe total message batches (entries) read from the storage for this broker (message batch per second).
pulsar_broker_msg_backlogGaugeThe total number of message backlogs in this broker (entries).

BookKeeper client metrics

All the BookKeeper client metrics are labeled with the following label:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
NameTypeDescription
pulsar_managedLedger_client_bookkeeper_client_BOOKIE_QUARANTINECounterThe number of bookie clients to be quarantined.

If you want to expose this metric, set bookkeeperClientExposeStatsToPrometheus to true in the broker.conf file.

Namespace metrics

Namespace metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to false.

All the namespace metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
NameTypeDescription
pulsar_topics_countGaugeThe number of Pulsar topics of the namespace owned by this broker.
pulsar_subscriptions_countGaugeThe number of Pulsar subscriptions of the namespace served by this broker.
pulsar_producers_countGaugeThe number of active producers of the namespace connected to this broker.
pulsar_consumers_countGaugeThe number of active consumers of the namespace connected to this broker.
pulsar_rate_inGaugeThe total message rate of the namespace coming into this broker (message per second).
pulsar_rate_outGaugeThe total message rate of the namespace going out from this broker (message per second).
pulsar_throughput_inGaugeThe total throughput of the namespace coming into this broker (byte per second).
pulsar_throughput_outGaugeThe total throughput of the namespace going out from this broker (byte per second).
pulsar_consumer_msg_ack_rateGaugeThe total message acknowledgment rate of the namespace owned by this broker (message per second).
pulsar_storage_sizeGaugeThe total storage size of the topics in this namespace owned by this broker (bytes).
pulsar_storage_logical_sizeGaugeThe storage size of topics in the namespace owned by the broker without replicas (in bytes).
pulsar_storage_backlog_sizeGaugeThe total backlog size of the topics of this namespace owned by this broker (in bytes).
pulsar_storage_offloaded_sizeGaugeThe total amount of the data in this namespace offloaded to the tiered storage (bytes).
pulsar_storage_write_rateGaugeThe total message batches (entries) written to the storage for this namespace (message batch per second).
pulsar_storage_read_rateGaugeThe total message batches (entries) read from the storage for this namespace (message batch per second).
pulsar_subscription_delayedGaugeThe total message batches (entries) are delayed for dispatching.
pulsarstorage_write_latency_leHistogramThe entry rate of a namespace that the storage write latency is smaller with a given threshold.
Available thresholds:
  • pulsar_storage_write_latency_le_0_5: <= 0.5ms
  • pulsar_storage_write_latency_le_1: <= 1ms
  • pulsar_storage_write_latency_le_5: <= 5ms
  • pulsar_storage_write_latency_le_10: <= 10ms
  • pulsar_storage_write_latency_le_20: <= 20ms
  • pulsar_storage_write_latency_le_50: <= 50ms
  • pulsar_storage_write_latency_le_100: <= 100ms
  • pulsar_storage_write_latency_le_200: <= 200ms
  • pulsar_storage_write_latency_le_1000: <= 1s
  • pulsar_storage_write_latency_le_overflow: > 1s
pulsarentry_size_leHistogramThe entry rate of a namespace that the entry size is smaller with a given threshold.
Available thresholds:
  • pulsar_entry_size_le_128: <= 128 bytes
  • pulsar_entry_size_le_512: <= 512 bytes
  • pulsar_entry_size_le_1_kb: <= 1 KB
  • pulsar_entry_size_le_2_kb: <= 2 KB
  • pulsar_entry_size_le_4_kb: <= 4 KB
  • pulsar_entry_size_le_16_kb: <= 16 KB
  • pulsar_entry_size_le_100_kb: <= 100 KB
  • pulsar_entry_size_le_1_mb: <= 1 MB
  • pulsar_entry_size_le_overflow: > 1 MB
pulsar_delayed_message_index_size_bytesGaugeThe total memory size allocated by DelayedDeliveryTracker of the namespace owned by this broker (in bytes).
pulsar_delayed_message_index_bucket_totalGaugeThe number of delayed message index buckets (immutable buckets + LastMutableBucket )
pulsar_delayed_message_index_loadedGaugeThe total number of delayed message indexes for in the memory.
pulsar_delayed_message_index_bucket_snapshot_size_bytesGaugeThe total size of delayed message index bucket snapshot (in bytes).
pulsar_delayed_message_index_bucket_op_countCounterThe total number of operation delayed message index bucket snapshots. The state label can be succeed,failed, andall (all means the total number of all states) and the type label can be create,load,delete, and merge.
pulsar_delayed_message_index_bucket_op_latency_msHistogramThe latency of delayed message index bucket snapshot operation with a given quantile (threshold). The type label can be create,load,delete, and merge
The label quantile can be:
  • quantile=”50” is operation latency between (0ms, 50ms]
  • quantile=”100” is operation latency between (50ms, 100ms]
  • quantile=”500” is operation latency between (100ms, 500ms]
  • quantile=”1000” is operation latency between (500ms, 1s]
  • quantile=”5000” is operation latency between (1s, 5s]
  • quantile=”30000” is operation latency between (5s, 30s]
  • quantile=”60000” is operation latency between (30s, 60s]
  • quantile=”overflow” is operation latency > 1m

Topic metrics

Topic metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the topic metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
  • topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
NameTypeDescription
pulsar_subscriptions_countGaugeThe number of Pulsar subscriptions of the topic served by this broker.
pulsar_producers_countGaugeThe number of active producers of the topic connected to this broker.
pulsar_consumers_countGaugeThe number of active consumers of the topic connected to this broker.
pulsar_rate_inGaugeThe total message rate of the topic coming into this broker (message per second).
pulsar_rate_outGaugeThe total message rate of the topic going out from this broker (message per second).
pulsar_publish_rate_limit_timesGaugeThe number of times the publish rate limit is triggered.
pulsar_throughput_inGaugeThe total throughput of the topic coming into this broker (byte per second).
pulsar_throughput_outGaugeThe total throughput of the topic going out from this broker (byte per second).
pulsar_consumer_msg_ack_rateGaugeThe total message acknowledgment rate of the topic connected to this broker (message per second).
pulsar_storage_sizeGaugeThe total storage size of the topics in this topic owned by this broker (bytes).
pulsar_storage_logical_sizeGaugeThe storage size of topics in the namespace owned by the broker without replicas (in bytes).
pulsar_storage_backlog_sizeGaugeThe total backlog size of the topics of this topic owned by this broker (in bytes).
pulsar_storage_offloaded_sizeGaugeThe total amount of the data in this topic offloaded to the tiered storage (bytes).
pulsar_storage_backlog_quota_limitGaugeThe total amount of the data in this topic that limit the backlog quota (bytes).
pulsar_storage_write_rateGaugeThe total message batches (entries) written to the storage for this topic (message batch per second).
pulsar_storage_read_rateGaugeThe total message batches (entries) read from the storage for this topic (message batch per second).
pulsar_subscription_delayedGaugeThe total message batches (entries) are delayed for dispatching.
pulsarstorage_write_latency_leHistogramThe entry rate of a topic that the storage write latency is smaller with a given threshold.
Available thresholds:
  • pulsar_storage_write_latency_le_0_5: <= 0.5ms
  • pulsar_storage_write_latency_le_1: <= 1ms
  • pulsar_storage_write_latency_le_5: <= 5ms
  • pulsar_storage_write_latency_le_10: <= 10ms
  • pulsar_storage_write_latency_le_20: <= 20ms
  • pulsar_storage_write_latency_le_50: <= 50ms
  • pulsar_storage_write_latency_le_100: <= 100ms
  • pulsar_storage_write_latency_le_200: <= 200ms
  • pulsar_storage_write_latency_le_1000: <= 1s
  • pulsar_storage_write_latency_le_overflow: > 1s
pulsarentry_size_leHistogramThe entry rate of a topic that the entry size is smaller with a given threshold.
Available thresholds:
  • pulsar_entry_size_le_128: <= 128 bytes
  • pulsar_entry_size_le_512: <= 512 bytes
  • pulsar_entry_size_le_1_kb: <= 1 KB
  • pulsar_entry_size_le_2_kb: <= 2 KB
  • pulsar_entry_size_le_4_kb: <= 4 KB
  • pulsar_entry_size_le_16_kb: <= 16 KB
  • pulsar_entry_size_le_100_kb: <= 100 KB
  • pulsar_entry_size_le_1_mb: <= 1 MB
  • pulsar_entry_size_le_overflow: > 1 MB
pulsar_in_bytes_totalCounterThe total number of messages in bytes received for this topic.
pulsar_in_messages_totalCounterThe total number of messages received for this topic.
pulsar_out_bytes_totalCounterThe total number of messages in bytes read from this topic.
pulsar_out_messages_totalCounterThe total number of messages read from this topic.
pulsar_compaction_removed_event_countGaugeThe total number of removed events of the compaction.
pulsar_compaction_succeed_countGaugeThe total number of successes of the compaction.
pulsar_compaction_failed_countGaugeThe total number of failures of the compaction.
pulsar_compaction_duration_time_in_millsGaugeThe duration time of the compaction.
pulsar_compaction_read_throughputGaugeThe read throughput of the compaction.
pulsar_compaction_write_throughputGaugeThe write throughput of the compaction.
pulsarcompaction_latency_le*HistogramThe compaction latency with given quantile.
Available thresholds:
  • pulsar_compaction_latency_le_0_5: <= 0.5ms
  • pulsar_compaction_latency_le_1: <= 1ms
  • pulsar_compaction_latency_le_5: <= 5ms
  • pulsar_compaction_latency_le_10: <= 10ms
  • pulsar_compaction_latency_le_20: <= 20ms
  • pulsar_compaction_latency_le_50: <= 50ms
  • pulsar_compaction_latency_le_100: <= 100ms
  • pulsar_compaction_latency_le_200: <= 200ms
  • pulsar_compaction_latency_le_1000: <= 1s
  • pulsar_compaction_latency_le_overflow: > 1s
pulsar_compaction_compacted_entries_countGaugeThe total number of the compacted entries.
pulsar_compaction_compacted_entries_sizeGaugeThe total size of the compacted entries.
pulsar_txn_tb_active_totalGaugeThe number of active transactions on this topic.
pulsar_txn_tb_aborted_totalCounterThe number of aborted transactions on the topic.
pulsar_txn_tb_committed_totalCounterThe number of committed transactions on the topic.
pulsar_delayed_message_index_size_bytesGaugeThe total memory size allocated by DelayedDeliveryTracker of the topic owned by this broker (in bytes).
pulsar_delayed_message_index_bucket_totalGaugeThe number of delayed message index buckets (immutable buckets + LastMutableBucket )
pulsar_delayed_message_index_loadedGaugeThe total number of delayed message indexes for in the memory.
pulsar_delayed_message_index_bucket_snapshot_size_bytesGaugeThe total size of delayed message index bucket snapshot (in bytes).
pulsar_delayed_message_index_bucket_op_countCounterThe total number of operation delayed message index bucket snapshots. The state label can be succeed,failed, andall (all means the total number of all states) and the type label can be create,load,delete, and merge.
pulsar_delayed_message_index_bucket_op_latency_msHistogramThe latency of delayed message index bucket snapshot operation with a given quantile (threshold). The labeltype label can be create,load,delete, and merge
The label quantile can be:
  • quantile=”50” is operation latency between (0ms, 50ms]
  • quantile=”100” is operation latency between (50ms, 100ms]
  • quantile=”500” is operation latency between (100ms, 500ms]
  • quantile=”1000” is operation latency between (500ms, 1s]
  • quantile=”5000” is operation latency between (1s, 5s]
  • quantile=”30000” is operation latency between (5s, 30s]
  • quantile=”60000” is operation latency between (30s, 60s]
  • quantile=”overflow” is operation latency > 1m

Replication metrics

If a namespace is configured to be replicated among multiple Pulsar clusters, the corresponding replication metrics is also exposed when replicationMetricsEnabled is enabled.

All the replication metrics are also labelled with remoteCluster=${pulsar_remote_cluster}.

NameTypeDescription
pulsar_replication_rate_inGaugeThe total message rate of the namespace replicating from remote cluster (messages/second).
pulsar_replication_rate_outGaugeThe total message rate of the namespace replicating to remote cluster (messages/second).
pulsar_replication_throughput_inGaugeThe total throughput of the namespace replicating from remote cluster (bytes/second).
pulsar_replication_throughput_outGaugeThe total throughput of the namespace replicating to remote cluster (bytes/second).
pulsar_replication_backlogGaugeThe total backlog of the namespace replicating to remote cluster (messages).
pulsar_replication_rate_expiredGaugeTotal rate of messages expired (messages/second).
pulsar_replication_connected_countGaugeThe count of replication-subscriber up and running to replicate to remote cluster.
pulsar_replication_delay_in_secondsGaugeTime in seconds from the time a message was produced to the time when it is about to be replicated.

Topic lookup metrics

NameTypeDescription
pulsar_broker_load_manager_bundle_assignmentGaugeThe summary of latency of bundles ownership operations.
pulsar_broker_lookupGaugeThe latency of all lookup operations.
pulsar_broker_lookup_redirectsGaugeThe number of lookup redirected requests.
pulsar_broker_lookup_answersGaugeThe number of lookup responses (i.e. not redirected requests).
pulsar_broker_lookup_failuresGaugeThe number of lookup failures.
pulsar_broker_lookup_pending_requestsGaugeThe number of pending lookups in broker. When it is up to the threshold, new requests are rejected.
pulsar_broker_topic_load_pending_requestsGaugeThe load of pending topic operations.

ManagedLedger metrics

All the managedLedger metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
  • quantile: quantile=${quantile}. Quantile is only for Histogram type metric, and represents the threshold for given Buckets.
NameTypeDescription
pulsar_ml_AddEntryBytesRateGaugeThe B/s rate of messages added
pulsar_ml_AddEntryWithReplicasBytesRateGaugeThe B/s rate of messages added with replicas
pulsar_ml_AddEntryErrorsGaugeThe number of addEntry requests that failed
pulsar_ml_AddEntryLatencyBucketsHistogramThe latency of adding a ledger entry with a given quantile (threshold), including time spent waiting in queue on the broker side.
Available quantile:
  • quantile=”0.0_0.5” is AddEntryLatency between (0.0ms, 0.5ms]
  • quantile=”0.5_1.0” is AddEntryLatency between (0.5ms, 1.0ms]
  • quantile=”1.0_5.0” is AddEntryLatency between (1ms, 5ms]
  • quantile=”5.0_10.0” is AddEntryLatency between (5ms, 10ms]
  • quantile=”10.0_20.0” is AddEntryLatency between (10ms, 20ms]
  • quantile=”20.0_50.0” is AddEntryLatency between (20ms, 50ms]
  • quantile=”50.0_100.0” is AddEntryLatency between (50ms, 100ms]
  • quantile=”100.0_200.0” is AddEntryLatency between (100ms, 200ms]
  • quantile=”200.0_1000.0” is AddEntryLatency between (200ms, 1s]
pulsar_ml_AddEntryLatencyBuckets_OVERFLOWGaugeThe number of times the AddEntryLatency is longer than 1 second
pulsar_ml_AddEntryMessagesRateGaugeThe msg/s rate of messages added
pulsar_ml_AddEntrySucceedGaugeThe number of addEntry requests that succeeded
pulsar_ml_EntrySizeBucketsHistogramThe added entry size of a ledger with a given quantile.
Available quantile:
  • quantile=”0.0_128.0” is EntrySize between (0byte, 128byte]
  • quantile=”128.0_512.0” is EntrySize between (128byte, 512byte]
  • quantile=”512.0_1024.0” is EntrySize between (512byte, 1KB]
  • quantile=”1024.0_2048.0” is EntrySize between (1KB, 2KB]
  • quantile=”2048.0_4096.0” is EntrySize between (2KB, 4KB]
  • quantile=”4096.0_16384.0” is EntrySize between (4KB, 16KB]
  • quantile=”16384.0_102400.0” is EntrySize between (16KB, 100KB]
  • quantile=”102400.0_1232896.0” is EntrySize between (100KB, 1MB]
pulsar_ml_EntrySizeBuckets_OVERFLOWGaugeThe number of times the EntrySize is larger than 1MB
pulsar_ml_LedgerSwitchLatencyBucketsHistogramThe ledger switch latency with a given quantile.
Available quantile:
  • quantile=”0.0_0.5” is EntrySize between (0ms, 0.5ms]
  • quantile=”0.5_1.0” is EntrySize between (0.5ms, 1ms]
  • quantile=”1.0_5.0” is EntrySize between (1ms, 5ms]
  • quantile=”5.0_10.0” is EntrySize between (5ms, 10ms]
  • quantile=”10.0_20.0” is EntrySize between (10ms, 20ms]
  • quantile=”20.0_50.0” is EntrySize between (20ms, 50ms]
  • quantile=”50.0_100.0” is EntrySize between (50ms, 100ms]
  • quantile=”100.0_200.0” is EntrySize between (100ms, 200ms]
  • quantile=”200.0_1000.0” is EntrySize between (200ms, 1000ms]
pulsar_ml_LedgerSwitchLatencyBuckets_OVERFLOWGaugeThe number of times the ledger switch latency is longer than 1 second
pulsar_ml_LedgerAddEntryLatencyBucketsHistogramThe latency for bookie client to persist a ledger entry from broker to BookKeeper service with a given quantile (threshold).
Available quantile:
  • quantile=”0.0_0.5” is LedgerAddEntryLatency between (0.0ms, 0.5ms]
  • quantile=”0.5_1.0” is LedgerAddEntryLatency between (0.5ms, 1.0ms]
  • quantile=”1.0_5.0” is LedgerAddEntryLatency between (1ms, 5ms]
  • quantile=”5.0_10.0” is LedgerAddEntryLatency between (5ms, 10ms]
  • quantile=”10.0_20.0” is LedgerAddEntryLatency between (10ms, 20ms]
  • quantile=”20.0_50.0” is LedgerAddEntryLatency between (20ms, 50ms]
  • quantile=”50.0_100.0” is LedgerAddEntryLatency between (50ms, 100ms]
  • quantile=”100.0_200.0” is LedgerAddEntryLatency between (100ms, 200ms]
  • quantile=”200.0_1000.0” is LedgerAddEntryLatency between (200ms, 1s]
pulsar_ml_LedgerAddEntryLatencyBuckets_OVERFLOWGaugeThe number of times the LedgerAddEntryLatency is longer than 1 second
pulsar_ml_MarkDeleteRateGaugeThe rate of mark-delete ops/s
pulsar_ml_NumberOfMessagesInBacklogGaugeThe number of backlog messages for all the consumers
pulsar_ml_ReadEntriesBytesRateGaugeThe B/s rate of messages read
pulsar_ml_ReadEntriesErrorsGaugeThe number of readEntries requests that failed
pulsar_ml_ReadEntriesRateGaugeThe msg/s rate of messages read
pulsar_ml_ReadEntriesSucceededGaugeThe number of readEntries requests that succeeded
pulsar_ml_StoredMessagesSizeGaugeThe total size of the messages in active ledgers (accounting for the multiple copies stored)

Managed cursor acknowledgment state

The acknowledgment state is persistent to the ledger first. When the acknowledgment state fails to be persistent to the ledger, they are persistent to ZooKeeper. To track the stats of acknowledgment, you can configure the metrics for the managed cursor.

All the cursor acknowledgment state metrics are labeled with the following labels:

  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

  • ledger_name: ledger_name=${pulsar_ledger_name}. ${pulsar_ledger_name} is the ledger name.

  • cursor_name: ledger_name=${pulsar_cursor_name}. ${pulsar_cursor_name} is the cursor name.

NameTypeDescription
pulsar_ml_cursor_persistLedgerSucceedGaugeThe number of acknowledgment states that is persistent to a ledger.
pulsar_ml_cursor_persistLedgerErrorsGaugeThe number of ledger errors occurred when acknowledgment states fail to be persistent to the ledger.
pulsar_ml_cursor_persistZookeeperSucceedGaugeThe number of acknowledgment states that is persistent to ZooKeeper.
pulsar_ml_cursor_persistZookeeperErrorsGaugeThe number of ledger errors occurred when acknowledgment states fail to be persistent to ZooKeeper.
pulsar_ml_cursor_nonContiguousDeletedMessagesRangeGaugeThe number of non-contiguous deleted messages ranges.
pulsar_ml_cursor_writeLedgerSizeGaugeThe size of write to ledger.
pulsar_ml_cursor_writeLedgerLogicalSizeGaugeThe size of write to ledger (accounting for without replicas).
pulsar_ml_cursor_readLedgerSizeGaugeThe size of read from ledger.

Subscription metrics

Subscription metrics are only exposed when exposeTopicLevelMetricsInPrometheus is set to true.

All the subscription metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
  • topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
  • subscription: subscription=${subscription}. ${subscription} is the topic subscription name.
NameTypeDescription
pulsar_subscription_back_logGaugeThe number of entries (messages/batched-messages) in unacknowledged state for a subscription
pulsar_subscription_back_log_no_delayedGaugeThe backlog of a subscription that does not contain the delay messages (entries).
pulsar_subscription_delayedGaugeThe total number of messages are delayed to be dispatched for a subscription (messages).
pulsar_subscription_msg_rate_redeliverGaugeThe total message rate for message being redelivered (message per second).
pulsar_subscription_unacked_messagesGaugeThe number of entries (messages/batched-messages) dispatched to consumers and are still unacknowledged
pulsar_subscription_blocked_on_unacked_messagesGaugeIndicate whether a subscription is blocked on unacknowledged messages or not.
  • 1 means the subscription is blocked on waiting for unacknowledged messages to be acked.
  • 0 means the subscription is not blocked on waiting for unacknowledged messages to be acked.
pulsar_subscription_msg_rate_outGaugeThe total message dispatch rate for a subscription (message per second).
pulsar_subscription_msg_throughput_outGaugeThe total message dispatch throughput for a subscription (byte per second).
pulsar_subscription_msg_ack_rateGaugeThe total message acknowledgment rate for a subscription (message per second).
pulsar_subscription_last_expire_timestampGaugeThe expiration timestamp of the last message on the subscription.
pulsar_subscription_last_acked_timestampGaugeThe timestamp of the last acknowledged message on the subscription.
pulsar_subscription_last_consumed_flow_timestampGaugeThe timestamp of the last received consume flow command on the subscription.
pulsar_subscription_last_consumed_timestampGaugeThe timestamp of the last consumed message on the subscription.
pulsar_subscription_last_mark_delete_advanced_timestampGaugeThe advanced timestamp of the last MarkDelete position on the subscription.
pulsar_subscription_msg_rate_expiredGaugeThe total rate of messages expired on this subscription (message per second).
pulsar_subscription_total_msg_expiredGaugeThe total number of messages expired on this subscription.
pulsar_subscription_msg_drop_rateGaugeThe rate of messages dropped on this subscription (message per second).
pulsar_subscription_consumers_countGaugeThe number of connected consumers on this subscription.
pulsar_subscription_filter_processed_msg_countCounterThe number of messages processed by EntryFilter.
pulsar_subscription_filter_accepted_msg_countCounterThe number of messages accepted by EntryFilter.
pulsar_subscription_filter_rejected_msg_countCounterThe number of messages rejected by EntryFilter.
pulsar_subscription_filter_rescheduled_msg_countCounterThe number of messages rescheduled by EntryFilter.
pulsar_delayed_message_index_size_bytesGaugeThe total memory size allocated by DelayedDeliveryTracker of the subscription owned by this broker (in bytes).
pulsar_delayed_message_index_bucket_totalGaugeThe number of delayed message index buckets (immutable buckets + LastMutableBucket )
pulsar_delayed_message_index_loadedGaugeThe total number of delayed message indexes for in the memory.
pulsar_delayed_message_index_bucket_snapshot_size_bytesGaugeThe total size of delayed message index bucket snapshot (in bytes).
pulsar_delayed_message_index_bucket_op_countCounterThe total number of operation delayed message index bucket snapshots. The state label can be succeed,failed, andall (all means the total number of all states) and the type label can be create,load,delete, and merge.
pulsar_delayed_message_index_bucket_op_latency_msHistogramThe latency of delayed message index bucket snapshot operation with a given quantile (threshold). The labeltype label can be create,load,delete, and merge
The label quantile can be:
  • quantile=”50” is operation latency between (0ms, 50ms]
  • quantile=”100” is operation latency between (50ms, 100ms]
  • quantile=”500” is operation latency between (100ms, 500ms]
  • quantile=”1000” is operation latency between (500ms, 1s]
  • quantile=”5000” is operation latency between (1s, 5s]
  • quantile=”30000” is operation latency between (5s, 30s]
  • quantile=”60000” is operation latency between (30s, 60s]
  • quantile=”overflow” is operation latency > 1m

Consumer metrics

Consumer metrics are only exposed when both exposeTopicLevelMetricsInPrometheus and exposeConsumerLevelMetricsInPrometheus are set to true.

All the consumer metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
  • topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
  • subscription: subscription=${subscription}. ${subscription} is the topic subscription name.
  • consumer_name: consumer_name=${consumer_name}. ${consumer_name} is the topic consumer name.
  • consumer_id: consumer_id=${consumer_id}. ${consumer_id} is the topic consumer id.
NameTypeDescription
pulsar_consumer_msg_rate_redeliverGaugeThe total message rate for message being redelivered (message per second).
pulsar_consumer_unacked_messagesGaugeThe total number of unacknowledged messages of a consumer (messages).
pulsar_consumer_blocked_on_unacked_messagesGaugeIndicate whether a consumer is blocked on unacknowledged messages or not.
  • 1 means the consumer is blocked on waiting for unacknowledged messages to be acked.
  • 0 means the consumer is not blocked on waiting for unacknowledged messages to be acked.
pulsar_consumer_msg_rate_outGaugeThe total message dispatch rate for a consumer (message per second).
pulsar_consumer_msg_ack_rateGaugeThe total rate of message ack (message per second).
pulsar_consumer_msg_throughput_outGaugeThe total message dispatch throughput for a consumer (byte per second).
pulsar_consumer_available_permitsGaugeThe available permits for for a consumer.

Managed ledger bookie client metrics

All the managed ledger bookie client metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
pulsarmanagedLedger_client_bookkeeper_ml_scheduler_completed_tasksGaugeThe number of tasks the scheduler executor execute completed.
The number of metrics determined by the scheduler executor thread number configured by managedLedgerNumSchedulerThreads in broker.conf.
pulsarmanagedLedger_client_bookkeeper_ml_scheduler_queueGaugeThe number of tasks queued in the scheduler executor’s queue.
The number of metrics determined by the scheduler executor’s thread number configured by managedLedgerNumSchedulerThreads in broker.conf.
pulsarmanagedLedger_client_bookkeeper_ml_scheduler_total_tasks*GaugeThe total number of tasks the scheduler executor received.
The number of metrics determined by the scheduler executor’s thread number configured by managedLedgerNumSchedulerThreads in broker.conf.
pulsar_managedLedger_client_bookkeeper_ml_scheduler_task_executionSummaryThe scheduler task execution latency calculated in milliseconds.
pulsar_managedLedger_client_bookkeeper_ml_scheduler_task_queuedSummaryThe scheduler task queued latency calculated in milliseconds.

Token metrics

All the token metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
pulsar_expired_token_totalCounterThe number of expired tokens in Pulsar.
pulsar_expiring_token_minutesHistogramThe remaining time of expiring tokens in minutes.

Authentication metrics

All the authentication metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • provider_name: provider_name=${provider_name}. ${provider_name} is the class name of the authentication provider.
  • auth_method: auth_method=${auth_method}. ${auth_method} is the authentication method of the authentication provider.
  • reason: reason=${reason}. ${reason} is the reason for failing authentication operation. (This label is only for pulsar_authentication_failures_total.)
NameTypeDescription
pulsar_authentication_success_totalCounterThe number of successful authentication operations.
pulsar_authentication_failures_totalCounterThe number of failing authentication operations.

Connection metrics

All the connection metrics are labelled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • broker: broker=${advertised_address}. ${advertised_address} is the advertised address of the broker.
  • metric: metric=${metric}. ${metric} is the connection metric collective name.
NameTypeDescription
pulsar_active_connectionsGaugeThe number of active connections.
pulsar_connection_created_total_countGaugeThe total number of connections.
pulsar_connection_create_success_countGaugeThe number of successfully created connections.
pulsar_connection_create_fail_countGaugeThe number of failed connections.
pulsar_connection_closed_total_countGaugeThe total number of closed connections.
pulsar_broker_throttled_connectionsGaugeThe number of throttled connections.
pulsar_broker_throttled_connections_global_limitGaugeThe number of throttled connections because of per-connection limit.

Jetty metrics

For a functions-worker running separately from brokers, its Jetty metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the jetty metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
jetty_requests_totalCounterNumber of requests.
jetty_requests_activeGaugeNumber of requests currently active.
jetty_requests_active_maxGaugeMaximum number of requests that have been active at once.
jetty_request_time_max_secondsGaugeMaximum time spent handling requests.
jetty_request_time_seconds_totalCounterTotal time spent in all request handling.
jetty_dispatched_totalCounterNumber of dispatches.
jetty_dispatched_activeGaugeNumber of dispatches currently active.
jetty_dispatched_active_maxGaugeMaximum number of active dispatches being handled.
jetty_dispatched_time_maxGaugeMaximum time spent in dispatch handling.
jetty_dispatched_time_seconds_totalCounterTotal time spent in dispatch handling.
jetty_async_requests_totalCounterTotal number of async requests.
jetty_async_requests_waitingGaugeCurrently waiting async requests.
jetty_async_requests_waiting_maxGaugeMaximum number of waiting async requests.
jetty_async_dispatches_totalCounterNumber of requests that have been asynchronously dispatched.
jetty_expires_totalCounterNumber of async requests that have expired.
jetty_responses_totalCounterNumber of responses, labeled by status code. The code label can be “1xx”, “2xx”, “3xx”, “4xx”, or “5xx”.
jetty_stats_secondsGaugeTime in seconds stats have been collected for.
jetty_responses_bytes_totalCounterTotal number of bytes across all responses.

Schema metrics

For a functions-worker running separately from brokers, its schema metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the schema metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
pulsar_schema_del_ops_failed_totalCounterNumber of failed operations to delete schemas.
pulsar_schema_get_ops_failed_totalCounterNumber of failed operations to get schemas.
pulsar_schema_put_ops_failed_totalCounterNumber of failed operations to send schemas.
pulsar_schema_compatible_totalCounterNumber of compatible schemas.
pulsar_schema_incompatible_totalCounterNumber of incompatible schemas.
pulsar_schema_del_ops_latencySummaryLatency of successful operations to delete schemas.
pulsar_schema_get_ops_latencySummaryLatency of successful operations to get schemas.
pulsar_schema_put_ops_latencySummaryLatency of successful operations to send schemas.

Offload metrics

All the offload metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
  • topic: topic=${pulsar_topic}. ${pulsar_topic} is the topic name.
NameTypeDescription
pulsar_ledgeroffloader_offload_errorCounterThe number of failed operations to offload.
pulsar_ledgeroffloader_offload_rateGaugeThe rate of offloading(byte per second).
pulsar_ledgeroffloader_read_offload_errorCounterThe number of failed operations to read offload ledgers.
pulsar_ledgeroffloader_read_offload_rateGaugeThe rate of reading entries from offload ledgers(byte per second).
pulsar_ledgeroffloader_write_storage_errorCounterThe number of failed operations to write to storage.
pulsar_ledgeroffloader_read_offload_index_latencySummaryThe latency of reading index from offload ledgers.
pulsar_ledgeroffloader_read_offload_data_latencySummaryThe latency of reading data from offload ledgers.
pulsar_ledgeroffloader_read_ledger_latencySummaryThe latency of reading entries from BookKeeper.
pulsar_ledgeroffloader_delete_offload_opsCounterThe total number of successful and failed operations to delete offload ledgers.

Web service executor metrics

For functions workers running separately from brokers, their Jetty metrics are only exposed when includeStandardPrometheusMetrics is set to true.

All the web service executor metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
NameTypeDescription
pulsar_web_executor_max_threadsGAUGEThe max threads of pulsar-web thread pool
pulsar_web_executor_min_threadsGAUGEThe min threads of pulsar-web thread pool
pulsar_web_executor_idle_threadsGAUGEThe idle threads of pulsar-web thread pool
pulsar_web_executor_active_threadsGAUGEThe number of threads performing tasks of pulsar-web thread pool
pulsar_web_executor_current_threadsGAUGEThe number of threads in the pulsar-web thread pool

Metadata store metrics

All the metadata store metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you configured in broker.conf.
  • name: name=${metadata-store|configuration-metadata-store|state-metadata-store}. ${name} is the metadata store name.
NameTypeDescription
pulsar_metadata_store_ops_latencyHistogramThe latency of getting/deleting/putting data from/to metadata store.
pulsar_metadata_store_put_bytes_totalCounterThe number of data put to metadata store.
pulsar_batch_metadata_store_executor_queue_sizeGaugeThe number of blocking operations in metadata store executor.
pulsar_batch_metadata_store_queue_wait_time_msHistogramThe waiting time of batch operations.
pulsar_batch_metadata_store_batch_execute_time_msHistogramThe duration of the batch execution in milliseconds.
pulsar_batch_metadata_store_batch_sizeHistogramThe number of read/write operations in the batch.

JVM Metrics

Process Metrics

NameTypeDescription
process_cpu_seconds_totalCounterTotal user and system CPU time spent in seconds.
process_start_time_secondsGaugeStart time of the process since unix epoch in seconds.
process_open_fdsGaugeNumber of open file descriptors.
process_max_fdsGaugeMaximum number of open file descriptors.
process_virtual_memory_bytesGaugeVirtual memory size in bytes.
process_resident_memory_bytesGaugeResident memory size in bytes.

Memory Metrics

NameTypeDescription
jvm_memory_objects_pending_finalizationGaugeThe number of objects waiting in the finalizer queue.
jvm_memory_bytes_usedGaugeUsed bytes of a given JVM memory area.
jvm_memory_bytes_committedGaugeCommitted (bytes) of a given JVM memory area.
jvm_memory_bytes_maxGaugeMax (bytes) of a given JVM memory area.
jvm_memory_bytes_initGaugeInitial bytes of a given JVM memory area.
jvm_memory_pool_bytes_usedUsed bytes of a given JVM memory pool.
jvm_memory_pool_bytes_committedGaugeCommitted bytes of a given JVM memory pool.
jvm_memory_pool_bytes_maxGaugeMax bytes of a given JVM memory pool.
jvm_memory_pool_bytes_initGaugeInitial bytes of a given JVM memory pool.
jvm_memory_pool_collection_used_bytesGaugeUsed bytes after the last collection of a given JVM memory pool.
jvm_memory_pool_collection_committed_bytesGaugeCommitted after last collection bytes of a given JVM memory pool.
jvm_memory_pool_collection_max_bytesGaugeMax bytes after the last collection of a given JVM memory pool.
jvm_memory_pool_collection_init_bytesGaugeInitial after last collection bytes of a given JVM memory pool.
jvm_memory_pool_allocated_bytes_totalCounterTotal bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.

Buffer Pools Metrics

NameTypeDescription
jvm_buffer_pool_used_bytesGaugeUsed bytes of a given JVM buffer pool.
jvm_buffer_pool_capacity_bytesGaugeBytes capacity of a given JVM buffer pool.
jvm_buffer_pool_used_buffersGaugeUsed buffers of a given JVM buffer pool.

Garbage Collectors Metrics

NameTypeDescription
jvm_gc_collection_secondsSummaryTime spent in a given JVM garbage collector in seconds.

Threads Metrics

NameTypeDescription
jvm_threads_currentGaugeCurrent thread count of a JVM
jvm_threads_daemonGaugeDaemon thread count of a JVM
jvm_threads_peakGaugePeak thread count of a JVM
jvm_threads_started_totalCounterStarted thread count of a JVM
jvm_threads_deadlockedGaugeCycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
jvm_threads_deadlocked_monitorGaugeCycles of JVM-threads that are in deadlock waiting to acquire object monitors
jvm_threads_stateGaugeCurrent count of threads by state

Classloaders Metrics

NameTypeDescription
jvm_classes_currently_loadedGaugeThe number of classes that are currently loaded in the JVM
jvm_classes_loaded_totalCounterThe total number of classes that have been loaded since the JVM has started execution
jvm_classes_unloaded_totalCounterThe total number of classes that have been unloaded since the JVM has started execution

Load balancing

This section shows all metrics related to broker load balancing.

Metrics - 图1note

  • Load balancing metrics are not exposed by default. If you want to access load balancing metrics, you need to expose them by setting the following configurations in the broker.conf or standalone.conf file and ensure that your cluster has an active producer or consumer.

    1. loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl
    2. loadBalancerEnabled=true
    3. exposeBundlesMetricsInPrometheus=true // Add this configuration to standalone.conf
  • Metrics with an asterisk (*) are only available in the extensible load balancer.

LoadBalancing metrics

All the loadbalancing metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • broker: broker=${broker}. ${broker} is the IP address of the broker
  • metric: metric=”loadBalancing”.
NameTypeDescription
pulsar_lb_bandwidth_in_usageGaugeThe broker inbound bandwidth usage (in percent).
pulsar_lb_bandwidth_out_usageGaugeThe broker outbound bandwidth usage (in percent).
pulsar_lb_cpu_usageGaugeThe broker cpu usage (in percent).
pulsar_lb_directMemory_usageGaugeThe broker process direct memory usage (in percent).
pulsar_lb_memory_usageGaugeThe broker process memory usage (in percent).
pulsar_lb_resource_usage {feature=max}GaugeThe max resource usage of the bandwidth, CPU, memory, and direct_memory.
pulsar_lb_resource_usage {feature=max_ema}GaugeThe broker load score (WeightedMaxEMA).

BundleUnloading metrics

All the bundleUnloading metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
  • metric: metric=”bundleUnloading”.
NameTypeDescription
pulsar_lb_unload_broker_totalCounterUnload broker count in this bundle unloading
pulsar_lb_unload_bundle_totalCounterBundle unload count in this bundle unloading. If the value of pulsar_lb_unload_bundle_total is greater than zero, it means that the bundle has been unloaded.
pulsar_lb_unload_broker_breakdown_total{result, reason}CounterUnload broker breakdown count grouped by result and reason labels.
pulsar_lb_resource_usage_stats{feature=max_ema, stat=avg}GaugeThe average of brokers’ load scores.
pulsar_lb_resource_usage_stats{feature=max_ema, stat=std}*GaugeThe standard deviation of brokers’ load scores.

BundleSplit metrics

All the bundleUnloading metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
  • metric: metric=”bundlesSplit”.
NameTypeDescription
pulsar_lb_bundles_split_totalCounterThe total count of bundle split in this leader broker
pulsar_lb_bundles_split_breakdown_total{result, reason}*CounterBundle split breakdown count grouped by the result and reason labels.

Bundle metrics

All the bundle metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • broker: broker=${broker}. ${broker} is the IP address of the broker
  • bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker
  • metric: metric=”bundle”.
NameTypeDescription
pulsar_bundle_msg_rate_inGaugeThe total message rate coming into the topics in this bundle (message per second).
pulsar_bundle_msg_rate_outGaugeThe total message rate going out from the topics in this bundle (message per second).
pulsar_bundle_topics_countGaugeThe topic count in this bundle.
pulsar_bundle_consumer_countGaugeThe consumer count of the topics in this bundle.
pulsar_bundle_producer_countGaugeThe producer count of the topics in this bundle.
pulsar_bundle_msg_throughput_inGaugeThe total throughput coming into the topics in this bundle (byte per second).
pulsar_bundle_msg_throughput_outGaugeThe total throughput going out from the topics in this bundle (byte per second).

Bundle assign metrics

All the bundle assign metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name you have configured in the broker.conf file.
  • broker: broker=${broker}. ${broker} is the IP address of the broker.
  • bundle: bundle=${bundle}. ${bundle} is the bundle range on this broker.
  • metric: metric=”assign”.
NameTypeDescription
pulsar_lb_assign_broker_breakdown_total{result, reason}*CounterAssign broker breakdown count grouped by result and reason labels.

Service unit state channel metrics

All the service unit state channel metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name you have configured in the broker.conf file.
  • metric: metric=”sunitStateChn”.
NameTypeDescription
pulsar_sunit_state_chn_owner_lookup_total{result, state}CounterThe owner broker lookup counts grouped by the result and state labels.
pulsar_sunit_state_chn_event_publish_ops_total{result, event}CounterThe published message count of service unit (e.g., bundle) state changes grouped by the result and event labels
pulsar_sunit_state_chn_subscribe_ops_total{result, event}CounterThe subscribed message count of service unit (e.g., bundle) state changes grouped by the result and event labels.
pulsar_sunit_state_chn_inactive_broker_cleanup_ops_total{result}CounterThe counts of inactive broker cleanup operations grouped by the result label.
pulsar_sunit_state_chn_orphan_su_cleanup_ops_totalCounterThe total count of orphan service unit (e.g., bundle) cleanup operations.
pulsar_sunit_state_chn_owned_su_totalGaugeThe number of owned bundles.
pulsar_sunit_state_chn_su_tombstone_cleanup_ops_totalCounterThe total count of deleted service units (e.g., bundles) tombstone operations.
pulsar_sunit_state_chn_cleanup_ops_total{result=Failure}CounterThe total count of cleanup operation failures.

Pulsar Functions

All the Pulsar Functions metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.
NameTypeDescription
pulsar_function_processed_successfully_totalCounterThe total number of messages processed successfully.
pulsar_function_processed_successfully_1min_totalCounterThe total number of messages processed successfully in the last 1 minute.
pulsar_function_system_exceptions_totalCounterThe total number of system exceptions.
pulsar_function_system_exceptions_1min_totalCounterThe total number of system exceptions in the last 1 minute.
pulsar_function_user_exceptions_totalCounterThe total number of user exceptions.
pulsar_function_user_exceptions_1min_totalCounterThe total number of user exceptions in the last 1 minute.
pulsar_function_process_latency_msSummaryThe process latency in milliseconds.
pulsar_function_process_latency_ms_1minSummaryThe process latency in milliseconds in the last 1 minute.
pulsar_function_last_invocationGaugeThe timestamp of the last invocation of the function.
pulsar_function_received_totalCounterThe total number of messages received from source.
pulsar_function_received_1min_totalCounterThe total number of messages received from source in the last 1 minute.
pulsarfunction_user_metricSummaryThe user-defined metrics.

Connectors

All the Pulsar connector metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • namespace: namespace=${pulsar_namespace}. ${pulsar_namespace} is the namespace name.

Connector metrics contain source metrics and sink metrics.

  • Source metrics

    NameTypeDescription
    pulsar_source_written_totalCounterThe total number of records written to a Pulsar topic.
    pulsar_source_written_1min_totalCounterThe total number of records written to a Pulsar topic in the last 1 minute.
    pulsar_source_received_totalCounterThe total number of records received from source.
    pulsar_source_received_1min_totalCounterThe total number of records received from source in the last 1 minute.
    pulsar_source_last_invocationGaugeThe timestamp of the last invocation of the source.
    pulsar_source_source_exceptionGaugeThe exception from a source.
    pulsar_source_source_exceptions_totalCounterThe total number of source exceptions.
    pulsar_source_source_exceptions_1min_totalCounterThe total number of source exceptions in the last 1 minute.
    pulsar_source_system_exceptionGaugeThe exception from system code.
    pulsar_source_system_exceptions_totalCounterThe total number of system exceptions.
    pulsar_source_system_exceptions_1min_totalCounterThe total number of system exceptions in the last 1 minute.
    pulsarsource_user_metricSummaryThe user-defined metrics.
  • Sink metrics

    NameTypeDescription
    pulsar_sink_written_totalCounterThe total number of records processed by a sink.
    pulsar_sink_written_1min_totalCounterThe total number of records processed by a sink in the last 1 minute.
    pulsar_sink_received_1min_totalCounterThe total number of messages that a sink has received from Pulsar topics in the last 1 minute.
    pulsar_sink_received_totalCounterThe total number of records that a sink has received from Pulsar topics.
    pulsar_sink_last_invocationGaugeThe timestamp of the last invocation of the sink.
    pulsar_sink_sink_exceptionGaugeThe exception from a sink.
    pulsar_sink_sink_exceptions_totalCounterThe total number of sink exceptions.
    pulsar_sink_sink_exceptions_1min_totalCounterThe total number of sink exceptions in the last 1 minute.
    pulsar_sink_system_exceptionGaugeThe exception from system code.
    pulsar_sink_system_exceptions_totalCounterThe total number of system exceptions.
    pulsar_sink_system_exceptions_1min_totalCounterThe total number of system exceptions in the last 1 minute.
    pulsarsink_user_metricSummaryThe user-defined metrics.

Proxy

All the proxy metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • kubernetes_pod_name: kubernetes_pod_name=${kubernetes_pod_name}. ${kubernetes_pod_name} is the Kubernetes pod name.
NameTypeDescription
pulsar_proxy_active_connectionsGaugeNumber of connections currently active in the proxy.
pulsar_proxy_new_connectionsCounterCounter of connections being opened in the proxy.
pulsar_proxy_rejected_connectionsCounterCounter for connections rejected due to throttling.
pulsar_proxy_binary_opsCounterCounter of proxy operations.
pulsar_proxy_binary_bytesCounterCounter of proxy bytes.

Pulsar SQL Worker

NameTypeDescription
split_bytes_readCounterNumber of bytes read from BookKeeper.
split_num_messages_deserializedCounterNumber of messages deserialized.
split_num_record_deserializedCounterNumber of records deserialized.
split_bytes_read_per_querySummaryTotal number of bytes read per query.
split_entry_deserialize_timeSummaryTime spent on deserializing entries.
split_entry_deserialize_time_per_querySummaryTime spent on deserializing entries per query.
split_entry_queue_dequeue_wait_timeSummaryTime spend on waiting to get entry from entry queue because it is empty.
split_entry_queue_dequeue_wait_time_per_querySummaryTotal time spent waiting to get entry from entry queue per query.
split_message_queue_dequeue_wait_time_per_querySummaryTime spent waiting to dequeue from message queue because is is empty per query.
split_message_queue_enqueue_wait_timeSummaryTime spent waiting for message queue enqueue because the message queue is full.
split_message_queue_enqueue_wait_time_per_querySummaryTime spent waiting for message queue enqueue because the message queue is full per query.
split_num_entries_per_batchSummaryNumber of entries per batch.
split_num_entries_per_querySummaryNumber of entries per query.
split_num_messages_deserialized_per_entrySummaryNumber of messages deserialized per entry.
split_num_messages_deserialized_per_querySummaryNumber of messages deserialized per query.
split_read_attemptsSummaryNumber of reading attempts (fail if queues are full).
split_read_attempts_per_querySummaryNumber of reading attempts per query.
split_read_latency_per_batchSummaryLatency of reads per batch.
split_read_latency_per_querySummaryTotal read latency per query.
split_record_deserialize_timeSummaryTime spent deserializing message to record. For example, Avro, JSON, and so on.
split_record_deserialize_time_per_querySummaryTime spent deserializing message to record per query.
split_total_execution_timeSummaryThe total execution time.

Pulsar transaction

All the transaction metrics are labeled with the following labels:

  • cluster: cluster=${pulsar_cluster}. ${pulsar_cluster} is the cluster name that you have configured in the broker.conf file.
  • coordinator_id: coordinator_id=${coordinator_id}. ${coordinator_id} is the coordinator id.
NameTypeDescription
pulsar_txn_active_countGaugeNumber of active transactions.
pulsar_txn_created_totalCounterNumber of created transactions.
pulsar_txn_committed_totalCounterNumber of committed transactions.
pulsar_txn_aborted_totalCounterNumber of aborted transactions of this coordinator.
pulsar_txn_timeout_totalCounterNumber of timeout transactions.
pulsar_txn_append_log_totalCounterNumber of append transaction logs.
pulsartxn_execution_latency_le*HistogramTransaction execution latency.
Available latencies are as below:
  • latency=”10” is TransactionExecutionLatency between (0ms, 10ms]
  • latency=”20” is TransactionExecutionLatency between (10ms, 20ms]
  • latency=”50” is TransactionExecutionLatency between (20ms, 50ms]
  • latency=”100” is TransactionExecutionLatency between (50ms, 100ms]
  • latency=”500” is TransactionExecutionLatency between (100ms, 500ms]
  • latency=”1000” is TransactionExecutionLatency between (500ms, 1000ms]
  • latency=”5000” is TransactionExecutionLatency between (1s, 5s]
  • latency=”15000” is TransactionExecutionLatency between (5s, 15s]
  • latency=”30000” is TransactionExecutionLatency between (15s, 30s]
  • latency=”60000” is TransactionExecutionLatency between (30s, 60s]
  • latency=”300000” is TransactionExecutionLatency between (1m,5m]
  • latency=”1500000” is TransactionExecutionLatency between (5m,15m]
  • latency=”3000000” is TransactionExecutionLatency between (15m,30m]
  • latency=”overflow” is TransactionExecutionLatency between (30m,∞]
pulsar_txn_tb_client_abort_failed_totalCounterThe number of failures to abort transactions for transaction buffer client.
pulsar_txn_tb_client_commit_failed_totalCounterThe number of failures to commit transaction for transaction buffer client.
pulsar_txn_tb_client_abort_latencySummaryThe latency of aborting transactions for transaction buffer client.
pulsar_txn_tb_client_commit_latencySummaryThe latency of committing transactions for transaction buffer client.
pulsar_txn_tb_client_pending_requestsCounterThe number of pending requests for transaction buffer client.
pulsar_txn_tp_committed_count_totalCounterThe number of committed transactions for pending ack store.
pulsar_txn_tp_aborted_count_totalCounterThe number of aborted transactions for pending ack store.
pulsar_txn_tp_commit_latencySummaryThe latency of committing transactions for transaction pending ack handle.