Configure Grafana Live HA setup

By default, Grafana Live uses in-memory data structures and in-memory PUB/SUB hub for handling subscriptions.

In a high availability Grafana setup involving several Grafana server instances behind a load balancer, you can find the following limitations:

  • Built-in features like dashboard change notifications will only be broadcasted to users connected to the same Grafana server process instance.
  • Streaming from Telegraf will deliver data only to clients connected to the same instance which received Telegraf data, active stream cache is not shared between different Grafana instances.
  • A separate unidirectional stream between Grafana and backend data source may be opened on different Grafana servers for the same channel.

To bypass these limitations, Grafana v8.1 has an experimental Live HA engine that requires Redis to work.

Configure Redis Live engine

When the Redis engine is configured, Grafana Live keeps its state in Redis and uses Redis PUB/SUB functionality to deliver messages to all subscribers throughout all Grafana server nodes.

Here is an example configuration:

  1. [live]
  2. ha_engine = redis
  3. ha_engine_address = 127.0.0.1:6379

For additional information, refer to the ha_engine and ha_engine_address options.

After running:

  • All built-in real-time notifications like dashboard changes are delivered to all Grafana server instances and broadcasted to all subscribers.
  • Streaming from Telegraf delivers messages to all subscribers.
  • A separate unidirectional stream between Grafana and backend data source opens on different Grafana servers. Publishing data to a channel delivers messages to instance subscribers, as a result, publications from different instances on different machines do not produce duplicate data on panels.

At the moment we only support single Redis node.

Note: It’s possible to use Redis Sentinel and Haproxy to achieve a highly available Redis setup. Redis nodes should be managed by Redis Sentinel to achieve automatic failover. Haproxy configuration example:

  1. listen redis
  2.   server redis-01 127.0.0.1:6380 check port 6380 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
  3.   server redis-02 127.0.0.1:6381 check port 6381 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 backup
  4.   bind *:6379
  5.   mode tcp
  6.   option tcpka
  7.   option tcplog
  8.   option tcp-check
  9.   tcp-check send PING\r\n
  10.   tcp-check expect string +PONG
  11.   tcp-check send info\ replication\r\n
  12.   tcp-check expect string role:master
  13.   tcp-check send QUIT\r\n
  14.   tcp-check expect string +OK
  15.   balance roundrobin

Next, point Grafana Live to Haproxy address:port.