您可以使用Pulsar的 admin API来创建和管理分区主题。

以下说明和命令中所使用 topic 的名称结构如下:

  1. persistent://tenant/namespace/topic

分区主题资源

创建

Partitioned topics in Pulsar must be explicitly created. When creating a new partitioned topic you need to provide a name for the topic as well as the desired number of partitions.

Note

By default, after 60 seconds of creation, topics are considered inactive and deleted automatically to prevent from generating trash data.

To disable this feature, set brokerDeleteInactiveTopicsEnabled to false.

To change the frequency of checking inactive topics, set brokerDeleteInactiveTopicsFrequencySeconds to your desired value.

For more information about these two parameters, see here.

pulsar-admin

你可以使用create-partitioned-topic命令创建partitioned topic,并指定topic的名字;使用-p--partitions标志指定分区数。

Here’s an example:

  1. $ bin/pulsar-admin topics create-partitioned-topic \
  2. persistent://my-tenant/my-namespace/my-topic \
  3. --partitions 4

Note

If there already exists a non partitioned topic with suffix ‘-partition-‘ followed by numeric value like ‘xyz-topic-partition-10’, then you can not create partitioned topic with name ‘xyz-topic’ as the partitions of the partitioned topic could override the existing non partitioned topic. You have to delete that non partitioned topic first then create the partitioned topic.

REST API

PUT /admin/v2/persistent/:tenant/:namespace/:topic/partitions

Java

  1. String topicName = "persistent://my-tenant/my-namespace/my-topic";
  2. int numPartitions = 4;
  3. admin.persistentTopics().createPartitionedTopic(topicName, numPartitions);

Create missed partitions

Try to create partitions for partitioned topic. The partitions of partition topic has to be created, can be used by repair partitions when topic auto creation is disabled

pulsar-admin

You can create missed partitions using the create-missed-partitions command and specifying the topic name as an argument.

Here’s an example:

  1. $ bin/pulsar-admin topics create-missed-partitions \
  2. persistent://my-tenant/my-namespace/my-topic \

REST API

POST /admin/v2/persistent/:tenant/:namespace/:topic

Java

  1. String topicName = "persistent://my-tenant/my-namespace/my-topic";
  2. admin.persistentTopics().createMissedPartitions(topicName);

获取元数据

Partitioned topics have metadata associated with them that you can fetch as a JSON object. The following metadata fields are currently available:

字段含义
partitionsTopic 分区的数量

pulsar-admin

You can see the number of partitions in a partitioned topic using the get-partitioned-topic-metadata subcommand. Here’s an example:

  1. $ pulsar-admin topics get-partitioned-topic-metadata \
  2. persistent://my-tenant/my-namespace/my-topic
  3. {
  4. "partitions": 4
  5. }

REST API

GET /admin/v2/persistent/:tenant/:namespace/:topic/partitions

Java

  1. String topicName = "persistent://my-tenant/my-namespace/my-topic";
  2. admin.persistentTopics().getPartitionedTopicMetadata(topicName);

更新

You can update the number of partitions on an existing partitioned topic if the topic is non-global. To update, the new number of partitions must be greater than the existing number.

Decrementing the number of partitions would deleting the topic, which is not supported in Pulsar.

已创建的分区生产者和消费者,将自动查找新创建的分区。

pulsar-admin

Partitioned topics can be updated using the update-partitioned-topic command.

  1. $ pulsar-admin topics update-partitioned-topic \
  2. persistent://my-tenant/my-namespace/my-topic \
  3. --partitions 8

REST API

POST /admin/v2/persistent/:tenant/:cluster/:namespace/:destination/partitions

Java

  1. admin.persistentTopics().updatePartitionedTopic(persistentTopic, numPartitions);

删除

pulsar-admin

Partitioned topics can be deleted using the delete-partitioned-topic command, specifying the topic by name:

  1. $ bin/pulsar-admin topics delete-partitioned-topic \
  2. persistent://my-tenant/my-namespace/my-topic

REST API

DELETE /admin/v2/persistent/:topic/:namespace/:destination/partitions

Java

  1. admin.persistentTopics().delete(persistentTopic);

获取资源列表

It provides a list of persistent topics existing under a given namespace.

pulsar-admin

  1. $ pulsar-admin topics list tenant/namespace
  2. persistent://tenant/namespace/topic1
  3. persistent://tenant/namespace/topic2

REST API

GET /admin/v2/persistent/:tenant/:namespace

Java

  1. admin.persistentTopics().getList(namespace);

统计信息

It shows current statistics of a given partitioned topic. Here’s an example payload:

  1. {
  2. "msgRateIn": 4641.528542257553,
  3. "msgThroughputIn": 44663039.74947473,
  4. "msgRateOut": 0,
  5. "msgThroughputOut": 0,
  6. "averageMsgSize": 1232439.816728665,
  7. "storageSize": 135532389160,
  8. "publishers": [
  9. {
  10. "msgRateIn": 57.855383881403576,
  11. "msgThroughputIn": 558994.7078932219,
  12. "averageMsgSize": 613135,
  13. "producerId": 0,
  14. "producerName": null,
  15. "address": null,
  16. "connectedSince": null
  17. }
  18. ],
  19. "subscriptions": {
  20. "my-topic_subscription": {
  21. "msgRateOut": 0,
  22. "msgThroughputOut": 0,
  23. "msgBacklog": 116632,
  24. "type": null,
  25. "msgRateExpired": 36.98245516804671,
  26. "consumers": []
  27. }
  28. },
  29. "replication": {}
  30. }

The following stats are available:

统计信息Description
msgRateInThe sum of all local and replication publishers’ publish rates in messages per second
msgThroughputInSame as msgRateIn but in bytes per second instead of messages per second
msgRateOutThe sum of all local and replication consumers’ dispatch rates in messages per second
msgThroughputOutSame as msgRateOut but in bytes per second instead of messages per second
averageMsgSizeAverage message size, in bytes, from this publisher within the last interval
storageSizeThe sum of the ledgers’ storage size for this topic
publishersThe list of all local publishers into the topic. There can be anywhere from zero to thousands.
producerIdInternal identifier for this producer on this topic
producerNameInternal identifier for this producer, generated by the client library
addressIP address and source port for the connection of this producer
connectedSinceTimestamp this producer was created or last reconnected
subscriptionsThe list of all local subscriptions to the topic
my-subscriptionThe name of this subscription (client defined)
msgBacklogThe count of messages in backlog for this subscription
msgBacklogNoDelayedThe count of messages in backlog without delayed messages for this subscription
typeThis subscription type
msgRateExpiredThe rate at which messages were discarded instead of dispatched from this subscription due to TTL
consumersThe list of connected consumers for this subscription
consumerNameInternal identifier for this consumer, generated by the client library
availablePermitsThe number of messages this consumer has space for in the client library’s listen queue. A value of 0 means the client library’s queue is full and receive() isn’t being called. 非零值意味着 consumer 可以接收消息。
replicationThis section gives the stats for cross-colo replication of this topic
replicationBacklogThe outbound replication backlog in messages
connectedWhether the outbound replicator is connected
replicationDelayInSecondsHow long the oldest message has been waiting to be sent through the connection, if connected is true
inboundConnectionThe IP and port of the broker in the remote cluster’s publisher connection to this broker
inboundConnectedSinceThe TCP connection being used to publish messages to the remote cluster. 如果没有连接到本地发布者,一分钟后连接将自动关闭。

pulsar-admin

The stats for the partitioned topic and its connected producers and consumers can be fetched by using the partitioned-stats command, specifying the topic by name:

  1. $ pulsar-admin topics partitioned-stats \
  2. persistent://test-tenant/namespace/topic \
  3. --per-partition

REST API

GET /admin/v2/persistent/:tenant/:namespace/:topic/partitioned-stats

Java

  1. admin.topics().getPartitionedStats(persistentTopic, true /* per partition */, false /* is precise backlog */);

Internal stats

获取 topic 的详细统计信息。

统计信息Description
entriesAddedCounterMessages published since this broker loaded this topic
numberOfEntriesTotal number of messages being tracked
totalSizeTotal storage size in bytes of all messages
currentLedgerEntriesCount of messages written to the ledger currently open for writing
currentLedgerSizeSize in bytes of messages written to ledger currently open for writing
lastLedgerCreatedTimestampTime when last ledger was created
lastLedgerCreationFailureTimestamptime when last ledger was failed
waitingCursorsCountHow many cursors are caught up and waiting for a new message to be published
pendingAddEntriesCountHow many messages have (asynchronous) write requests we are waiting on completion
lastConfirmedEntryThe ledgerid:entryid of the last message successfully written. 如果 entryid 为 -1,则 ledger 已经允许写入或正在开放写入权限,但还没有写入 entry。
stateThe state of the cursor ledger. Open means we have a cursor ledger for saving updates of the markDeletePosition.
ledgersThe ordered list of all ledgers for this topic holding its messages
cursorsThe list of all cursors on this topic. There will be one for every subscription you saw in the topic stats.
markDeletePositionThe ack position: the last message the subscriber acknowledged receiving
readPositionThe latest position of subscriber for reading message
waitingReadOpThis is true when the subscription has read the latest message published to the topic and is waiting on new messages to be published.
pendingReadOpsThe counter for how many outstanding read requests to the BookKeepers we have in progress
messagesConsumedCounterNumber of messages this cursor has acked since this broker loaded this topic
cursorLedgerThe ledger being used to persistently store the current markDeletePosition
cursorLedgerLastEntryThe last entryid used to persistently store the current markDeletePosition
individuallyDeletedMessagesIf Acks are being done out of order, shows the ranges of messages Acked between the markDeletePosition and the read-position
lastLedgerSwitchTimestampThe last time the cursor ledger was rolled over
  1. {
  2. "entriesAddedCounter": 20449518,
  3. "numberOfEntries": 3233,
  4. "totalSize": 331482,
  5. "currentLedgerEntries": 3233,
  6. "currentLedgerSize": 331482,
  7. "lastLedgerCreatedTimestamp": "2016-06-29 03:00:23.825",
  8. "lastLedgerCreationFailureTimestamp": null,
  9. "waitingCursorsCount": 1,
  10. "pendingAddEntriesCount": 0,
  11. "lastConfirmedEntry": "324711539:3232",
  12. "state": "LedgerOpened",
  13. "ledgers": [
  14. {
  15. "ledgerId": 324711539,
  16. "entries": 0,
  17. "size": 0
  18. }
  19. ],
  20. "cursors": {
  21. "my-subscription": {
  22. "markDeletePosition": "324711539:3133",
  23. "readPosition": "324711539:3233",
  24. "waitingReadOp": true,
  25. "pendingReadOps": 0,
  26. "messagesConsumedCounter": 20449501,
  27. "cursorLedger": 324702104,
  28. "cursorLedgerLastEntry": 21,
  29. "individuallyDeletedMessages": "[(324711539:3134‥324711539:3136], (324711539:3137‥324711539:3140], ]",
  30. "lastLedgerSwitchTimestamp": "2016-06-29 01:30:19.313",
  31. "state": "Open"
  32. }
  33. }
  34. }

pulsar-admin

The internal stats for the partitioned topic can be fetched by using the stats-internal command, specifying the topic by name:

  1. $ pulsar-admin topics stats-internal \
  2. persistent://test-tenant/namespace/topic

REST API

GET /admin/v2/persistent/:tenant/:namespace/:topic/internalStats

Java

  1. admin.persistentTopics().getInternalStats(persistentTopic);