Monitoring

To monitor the NATS Streaming system, a lightweight HTTP server is used on a dedicated monitoring port. The monitoring server provides several endpoints, all returning a JSON object.

Enabling from the command line

To enable the monitoring server, start the NATS Streaming Server with the monitoring flag -m (or -ms) and specify the monitoring port.

Monitoring options

  1. -m, --http_port PORT             HTTP PORT for monitoring
  2. -ms,--https_port PORT            Use HTTPS PORT for monitoring (requires TLS cert and key)

To enable monitoring via the configuration file, use http: "host:port" or https: "host:port". There is no explicit configuration flag for the monitoring interface.

For example, after running this:

  1. nats-streaming-server -m 8222

you should see that the NATS Streaming server starts with the HTTP monitoring port enabled:

  1. [19339] 2019/06/24 15:02:38.251091 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.15.1
  2. [19339] 2019/06/24 15:02:38.251238 [INF] STREAM: ServerID: 0Z2HXClEM6BPsGaKcoHg5N
  3. [19339] 2019/06/24 15:02:38.251243 [INF] STREAM: Go version: go1.12
  4. [19339] 2019/06/24 15:02:38.251862 [INF] Starting nats-server version 2.0.0
  5. [19339] 2019/06/24 15:02:38.251873 [INF] Git commit [not set]
  6. [19339] 2019/06/24 15:02:38.252173 [INF] Starting http monitor on 0.0.0.0:8222
  7. [19339] 2019/06/24 15:02:38.252248 [INF] Listening for client connections on 0.0.0.0:4222
  8. (...)

You can then point your browser (or curl) to http://localhost:8222/streaming

Enabling from the configuration file

To start via the configuration file you can define the monitoring port as follows:

  1. http_port = 8222

Then use the -sc flag to customize the NATS Streaming configuration:

  1. nats-streaming-server -sc nats-streaming.conf -ns nats://demo.nats.io:4222 -SDV

Confirm that the monitoring endpoint is enabled by sending a request:

  1. curl 127.0.0.1:8222/streaming/channelsz
  1. {
  2.   "cluster_id": "test-cluster",
  3.   "server_id": "dXUsNRef1z25NpcFmZhBNj",
  4.   "now": "2019-06-24T15:18:37.388938-07:00",
  5.   "offset": 0,
  6.   "limit": 1024,
  7.   "count": 0,
  8.   "total": 0
  9. }

Monitoring a NATS Streaming channel with Grafana and Prometheus

Here you’ll find examples demonstrating how to use Prometheus query expressions to monitor NATS streaming channels.

Pending Messages from Channel Foo

  1. sum(nss_chan_subs_pending_count{channel="foo"}) by (client_id)

Pending

Messages Per Sec Delivered on Channel Foo

In this case, 3 is the size of the quorum of NATS Streaming Server nodes. In case of a single instance backed by a relational database we would set it to 1:

  1. sum(rate(nss_chan_msgs_total{channel="foo"}[5m])) by (channel) / 3

msgs-per-sec

Msgs/Sec vs Pending on Channel

Example of combining the rate of messages with the pending count to detect whether processing is getting behind:

  1. sum(rate(nss_chan_msgs_total{channel="foo"}[5m])) by (channel) / 3
  2. sum(nss_chan_subs_pending_count{channel="foo"}) by (channel) / 3

combination