System Events

NATS servers leverage Accounts support and generate events such as:

  • account connect/disconnect
  • authentication errors
  • server shutdown
  • server stat summary

In addition the server supports a limited number of requests that can be used to query for account connections, server stat summaries, and pinging servers in the cluster.

These events are enabled by configuring system_account and subscribing/requesting using a system account user.

Accounts are used so that subscriptions from your applications, say >, do not receive system events and vice versa. Using accounts requires either:

  • Configuring authentication locally and listing one of the accounts in system_account
  • Or by using decentralized authentication and authorization via jwt as shown in this Tutorial. In this case system_account contains the account public key.

Available Events and Services

The system account publishes messages under well known subject patterns.

Server initiated events:

  • $SYS.ACCOUNT.<id>.CONNECT (client connects)
  • $SYS.ACCOUNT.<id>.DISCONNECT (client disconnects)
  • $SYS.SERVER.ACCOUNT.<id>.CONNS (connections for an account changed)
  • $SYS.SERVER.<id>.CLIENT.AUTH.ERR (authentication error)
  • $SYS.ACCOUNT.<id>.LEAFNODE.CONNECT (leaf node connnects)
  • $SYS.ACCOUNT.<id>.LEAFNODE.DISCONNECT (leaf node disconnects)
  • $SYS.SERVER.<id>.STATSZ (stats summary)

In addition other tools with system account privileges, can initiate requests (Examples can be found here):

  • $SYS.REQ.SERVER.<id>.STATSZ (request server stat summary)
  • $SYS.REQ.SERVER.PING (discover servers - will return multiple messages)

Monitoring endpoints as listed in the table below are accessible as system services using the following subject pattern:

  • $SYS.REQ.SERVER.<id>.<endpoint-name> (request server monitoring endpoint corresponding to endpoint name.)
  • $SYS.REQ.SERVER.PING.<endpoint-name> (from all server, request server monitoring endpoint corresponding to endpoint name - will return multiple messages)
Endpoint Endpoint Name
General Server Information VARZ
Connections CONNZ
Routing ROUTEZ
Gateways GATEWAYZ
Leaf Nodes LEAFZ
Subscription Routing SUBSZ
JetStream JSZ
Accounts ACCOUNTZ
  • "$SYS.REQ.ACCOUNT.<account-id>.<endpoint-name>(from all server, request account specific monitoring endpoint corresponding to account id and endpoint name - will return multiple messages)
Endpoint Endpoint Name
Connections CONNZ
Leaf Nodes LEAFZ
Subscription Routing SUBSZ
JetStream JSZ
Account INFO

Servers like nats-account-server publish system account messages when a claim is updated, the nats-server listens for them, and updates its account information accordingly:

  • $SYS.ACCOUNT.<id>.CLAIMS.UPDATE

With these few messages you can build useful monitoring tools:

  • health/load of your servers
  • client connects/disconnects
  • account connections
  • authentication errors

Local Configuration

To make use of System events, just using accounts, your configuration can look like this:

  1. accounts: {
  2. USERS: {
  3. users: [
  4. {user: a, password: a}
  5. ]
  6. },
  7. SYS: {
  8. users: [
  9. {user: admin, password: changeit}
  10. ]
  11. },
  12. }
  13. system_account: SYS

Please note that applications now have to authenticate such that a connection can be associated with an account. In this example username and password were chosen for simplicity of the demonstration. Subscribe to all system events like this nats-sub -s nats://admin:changeit@localhost:4222 ">" and observe what happens when you do something like nats-pub -s "nats://a:a@localhost:4222" foo bar. Examples on how to use system services can be found here.