About alerting high availability

The Grafana Alerting system has two main components: a Scheduler and an internal Alertmanager. The Scheduler evaluates your alert rules, while the internal Alertmanager manages routing and grouping.

When running Grafana Alerting in high availability, the operational mode of the scheduler remains unaffected, and each Grafana instance evaluates all alerts. The operational change happens in the Alertmanager when it deduplicates alert notifications across Grafana instances.

High availability High availability Alerting high availability - 图2

The coordination between Grafana instances happens via a Gossip protocol. Alerts are not gossiped between instances and each scheduler delivers the same volume of alerts to each Alertmanager.

The two types of messages gossiped between Grafana instances are:

  • Notification logs: Who (which instance) notified what (which alert).
  • Silences: If an alert should fire or not.

The notification logs and silences are persisted in the database periodically and during a graceful Grafana shut down.

For configuration instructions, refer to enable alerting high availability.