In Rancher 2.5, the notifier application was improved. There are now two ways to enable notifiers. The older way is documented in this section, and the new application for notifiers is documented here.

Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action.

Rancher integrates with a variety of popular IT services, including:

  • Slack: Send alert notifications to your Slack channels.
  • Email: Choose email recipients for alert notifications.
  • PagerDuty: Route notifications to staff by phone, SMS, or personal email.
  • WebHooks: Update a webpage with alert notifications.
  • WeChat: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts.
  • DingTalk: (Available as of v2.4.6) Send alert notifications to DingTalk using a webhook.
  • Microsoft Teams: (Available as of v2.4.6) Send alert notifications to Teams using a webhook.

This section covers the following topics:

Roles-based Access Control for Notifiers

Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don’t need to dispense privileges like SMTP server access or cloud account access.

Adding Notifiers

Set up a notifier so that you can begin configuring and sending alerts.

  1. From the Global View, open the cluster that you want to add a notifier.
  2. From the main menu, select Tools > Notifiers. Then click Add Notifier.
  3. Select the service you want to use as your notifier, and then fill out the form. For help filling out the form, refer to the configuration section below.
  4. Click Test. You should receive a notification confirming that the notifier is configured correctly.
  5. Click Add to complete adding the notifier.

Result: Your notifier is added to Rancher.

Configuration

Slack

FieldExplanation
NameEnter a Name for the notifier.
URLFrom Slack, create a webhook. For instructions, see the Slack Documentation. Then enter the Slack webhook URL.
Default ChannelEnter the name of the channel that you want to send alert notifications in the following format: #<channelname>. Both public and private channels are supported.
Proxy URLProxy for the Slack webhook.
Send Resolved AlertsAvailable as of v2.3.0 Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, the Slack channel you’re configuring for the notifier outputs Slack setting validated.

Email

FieldExplanation
NameEnter a Name for the notifier.
Default Recipient AddressEnter the email address that you want to receive the notification.
Send Resolved AlertsAvailable as of v2.3.0 Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

SMTP Server Configuration:

FieldExplanation
SenderEnter an email address available on your mail server that you want to send the notification.
HostEnter the IP address or hostname for your SMTP server. Example: smtp.email.com
PortIn the Port field, enter the port used for email. Typically, TLS uses 587 and SSL uses 465.
Use TLSIf you’re using TLS, make sure Use TLS is selected.
UsernameUsername to authenticate with the SMTP server.
PasswordPassword to authenticate with the SMTP server.

Validation: Click Test. If the test is successful, Rancher prints settings validated and you receive a test notification email.

PagerDuty

FieldExplanation
NameEnter a Name for the notifier.
Default Integration KeyFrom PagerDuty, create a Prometheus integration. For instructions, see the PagerDuty Documentation. Then enter the integration key.
Service KeyThe same as the integration key. For instructions on creating a Prometheus integration, see the PagerDuty Documentation. Then enter the integration key.
Send Resolved AlertsAvailable as of v2.3.0 Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, your PagerDuty endpoint outputs PagerDuty setting validated.

Webhook

FieldExplanation
NameEnter a Name for the notifier.
URLUsing the app of your choice, create a webhook URL.
Proxy URLProxy for the webhook.
Send Resolved AlertsAvailable as of v2.3.0 Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, the URL you’re configuring as a notifier outputs Webhook setting validated.

WeChat

Available as of v2.2.0

FieldExplanation
NameEnter a Name for the notifier.
Corporation IDEnter the “EnterpriseID” of your corporation. You can get it fro the Profile page.
Application Agent IDFrom Enterprise WeChat, create an application in the Application page, and then enter the “AgentId” of this application. You will also need to enter the application secret.
Application SecretThe secret that corresponds to the Application Agent ID.
Recipient TypeParty, tag, or user.
Default RecipientThe default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from Contacts page.
Proxy URLIf you are using a proxy, enter the proxy URL.
Send Resolved AlertsAvailable as of v2.3.0 Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, you should receive an alert message.

DingTalk

Available as of v2.4.6

FieldExplanation
NameEnter a Name for the notifier.
Webhook URLEnter the DingTalk webhook URL. For help setting up the webhook, refer to the DingTalk documentation.
SecretOptional: Enter a secret for the DingTalk webhook.
Proxy URLOptional: Enter a proxy for the DingTalk webhook.
Send Resolved AlertsWhether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, the DingTalk notifier output is DingTalk setting validated.

Microsoft Teams

Available as of v2.4.6

FieldExplanation
NameEnter a Name for the notifier.
Webhook URLEnter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the Teams Documentation.
Proxy URLOptional: Enter a proxy for the Teams webhook.
Send Resolved AlertsWhether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage)

Validation: Click Test. If the test is successful, the Teams notifier output is MicrosoftTeams setting validated.

Managing Notifiers

After you set up notifiers, you can manage them. From the Global view, open the cluster that you want to manage your notifiers. Select Tools > Notifiers. You can:

  • Edit their settings that you configured during their initial setup.
  • Clone them, to quickly setup slightly different notifiers.
  • Delete them when they’re no longer necessary.

Example Payload for a Webhook Alert Notifier

  1. {
  2. "receiver": "c-2a3bc:kube-components-alert",
  3. "status": "firing",
  4. "alerts": [
  5. {
  6. "status": "firing",
  7. "labels": {
  8. "alert_name": "Scheduler is unavailable",
  9. "alert_type": "systemService",
  10. "cluster_name": "mycluster (ID: c-2a3bc)",
  11. "component_name": "scheduler",
  12. "group_id": "c-2a3bc:kube-components-alert",
  13. "logs": "Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused",
  14. "rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service",
  15. "severity": "critical"
  16. },
  17. "annotations": {},
  18. "startsAt": "2020-01-30T19:18:13.321684733Z",
  19. "endsAt": "0001-01-01T00:00:00Z",
  20. "generatorURL": ""
  21. }
  22. ],
  23. "groupLabels": {
  24. "component_name": "scheduler",
  25. "rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service"
  26. },
  27. "commonLabels": {
  28. "alert_name": "Scheduler is unavailable",
  29. "alert_type": "systemService",
  30. "cluster_name": "mycluster (ID: c-2a3bc)"
  31. }
  32. }

What’s Next?

After creating a notifier, set up alerts to receive notifications of Rancher system events.