Manage multi-tenant notifications with Notification Manager

Notification Manager manages notifications in KubeSphere. It receives alerts or notifications from different senders and then send notifications to different users.

Supported senders includes:

  • Prometheus Alertmanager
  • Custom sender (Coming soon)

Supported receivers includes:

Notification Manager

QuickStart

Config Prometheus Alertmanager to send alerts to Notification Manager

Notification Manager uses port 19093 and API path /api/v2/alerts to receive alerts sent from Prometheus Alertmanager of Kubesphere.

To receive Alertmanager alerts, KubeSphere already added Alertmanager webhook and route configurations like below ( by editing the Secret alertmanager-main in the namespace kubesphere-monitoring-system ):

Send Prometheus alerts to Notification Manager:

  1. "receivers":
  2. - "name": "prometheus"
  3. "webhook_configs":
  4. - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5. "route":
  6. "routes":
  7. - "match":
  8. "alerttype": ""
  9. "receiver": "prometheus"

Send event alerts to Notification Manager:

  1. "receivers":
  2. - "name": "event"
  3. "webhook_configs":
  4. - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5. "send_resolved": false
  6. "route":
  7. "routes":
  8. - "match":
  9. "alerttype": "event"
  10. "receiver": "event"
  11. "group_interval": "30s"

Send auditing alerts to Notification Manager:

  1. "receivers":
  2. - "name": "auditing"
  3. "webhook_configs":
  4. - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
  5. "send_resolved": false
  6. "route":
  7. "routes":
  8. - "match":
  9. "alerttype": "auditing"
  10. "receiver": "auditing"
  11. "group_interval": "30s"

Note

The above is the default configuration. If you do not want to receive a certain type of alert, you can delete the corresponding configuration.

Configure receivers

Notification Manager now supports three types of receivers: Email, WeChat Work and Slack. Only the administrator can configure receivers.

Email

If a tenant named test-user who wants to receive notifications from email, just create an email receiver like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. password: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: test-user-email-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: EmailConfig
  15. metadata:
  16. labels:
  17. app: notification-manager
  18. type: tenant
  19. user: test-user
  20. name: test-user-config
  21. namespace: kubesphere-monitoring-system
  22. spec:
  23. authPassword:
  24. key: password
  25. name: test-user-email-secret
  26. authUsername: abc1
  27. from: [email protected]
  28. requireTLS: true
  29. smartHost:
  30. host: imap.xyz.com
  31. port: "25"
  32. ---
  33. apiVersion: notification.kubesphere.io/v1alpha1
  34. kind: EmailReceiver
  35. metadata:
  36. labels:
  37. app: notification-manager
  38. type: tenant
  39. user: test-user
  40. name: test-user-receiver
  41. namespace: kubesphere-monitoring-system
  42. spec:
  43. emailConfigSelector:
  44. matchLabels:
  45. type: tenant
  46. user: test-user
  47. to:
  48. - [email protected]
  49. - [email protected]
  50. EOF

The emailConfigSelector is a selector to select EmailConfig for email receiver, if the emailConfigSelector is not set, receiver will use the default email config. You can create a default email config like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. password: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: default-email-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: EmailConfig
  15. metadata:
  16. labels:
  17. app: notification-manager
  18. type: default
  19. name: default-email-config
  20. namespace: kubesphere-monitoring-system
  21. spec:
  22. authPassword:
  23. key: password
  24. name: default-email-secret
  25. authUsername: default
  26. from: [email protected]
  27. requireTLS: true
  28. smartHost:
  29. host: imap.xyz.com
  30. port: "25"
  31. EOF

Email receivers with label type: tenant only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global email receiver with label type: global as below::

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: EmailReceiver
  4. metadata:
  5. labels:
  6. app: notification-manager
  7. type: global
  8. name: global-email-receiver
  9. namespace: kubesphere-monitoring-system
  10. spec:
  11. to:
  12. - [email protected]
  13. EOF

Note

Global email receiver will use the default email config.

Wechat Work

Notification Manager supports sending notification to Wechat Work. If a tenant named test-user who wants to receive notifications from Wechat Work, just create a wechat receiver like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. wechat: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: test-user-wechat-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: WechatConfig
  15. metadata:
  16. name: test-user-config
  17. namespace: kubesphere-monitoring-system
  18. labels:
  19. app: notification-manager
  20. type: tenant
  21. user: test-user
  22. spec:
  23. wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  24. wechatApiSecret:
  25. key: wechat
  26. name: test-user-wehat-secret
  27. wechatApiCorpId: wwfd76b24f06513578
  28. wechatApiAgentId: "1000002"
  29. ---
  30. apiVersion: notification.kubesphere.io/v1alpha1
  31. kind: WechatReceiver
  32. metadata:
  33. name: test-user-wechat
  34. namespace: kubesphere-monitoring-system
  35. labels:
  36. app: notification-manager
  37. type: tenant
  38. user: test-user
  39. spec:
  40. wechatConfigSelector:
  41. matchLabels:
  42. type: tenant
  43. user: test-user
  44. # optional
  45. # One of toUser, toParty, toParty should be specified.
  46. toUser: user1 | user2
  47. toParty: party1 | party2
  48. toTag: tag1 | tag2
  49. EOF

Info

  • wechatApiCorpId is the id of your Wechat Work.
  • wechatApiAgentId is the id of app sending message to user in your Wechat Work
  • wechatApiSecret is the secret of this app, you can get these two parameters in App Management of your Wechat Work.
  • Any user, party or tag who wants to receive notifications must be in the allowed users list of this app.

The wechatConfigSelector is a selector to select WechatConfig for wechat receiver, if the wechatConfigSelector is not set, wechat receiver will use the default wechat config. You can create a default wechat config like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. wechat: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: default-wechat-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: WechatConfig
  15. metadata:
  16. name: default-wechat-config
  17. namespace: kubesphere-monitoring-system
  18. labels:
  19. app: notification-manager
  20. type: default
  21. spec:
  22. wechatApiUrl: https://qyapi.weixin.qq.com/cgi-bin/
  23. wechatApiSecret:
  24. key: wechat
  25. name: default-wechat-secret
  26. wechatApiCorpId: wwfd76b24f06513578
  27. wechatApiAgentId: "1000002"
  28. EOF

Wechat receivers with label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global wechat receiver with label type: global as below:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: WechatReceiver
  4. metadata:
  5. name: global-wechat-wechat
  6. namespace: kubesphere-monitoring-system
  7. labels:
  8. app: notification-manager
  9. type: global
  10. spec:
  11. # optional
  12. # One of toUser, toParty, toParty should be specified.
  13. toUser: global
  14. toParty: global
  15. toTag: global
  16. EOF

Note

Global wechat receiver will use the default wechat config.

Slack

Notification Manager supports sending notification to slack channels. If a tenant named test-user who wants to receive notifications from slack, just create a slack receiver like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. token: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: test-user-slack-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: SlackConfig
  15. metadata:
  16. name: test-user-config
  17. namespace: kubesphere-monitoring-system
  18. labels:
  19. app: notification-manager
  20. type: tenant
  21. user: test-user
  22. spec:
  23. slackTokenSecret:
  24. key: token
  25. name: test-user-slack-secret
  26. ---
  27. apiVersion: notification.kubesphere.io/v1alpha1
  28. kind: SlackReceiver
  29. metadata:
  30. name: test-user-slack
  31. namespace: kubesphere-monitoring-system
  32. labels:
  33. app: notification-manager
  34. type: tenant
  35. user: test-user
  36. spec:
  37. slackConfigSelector:
  38. matchLabels:
  39. type: tenant
  40. user: test-user
  41. channel: alert
  42. EOF

Info

  • Slack token is the OAuth Access Token or Bot User OAuth Access Token when you create a slack app.
  • This app must have the scope chat:write.
  • The user who creates the app or bot user must be in the channel to which you want to send notifications.

The slackConfigSelector is a selector to select SlackConfig for slack receiver, if the slackConfigSelector is not set, slack receiver will use the default slack config. You can create a default slack config like this.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. data:
  4. token: dGVzdA==
  5. kind: Secret
  6. metadata:
  7. labels:
  8. app: notification-manager
  9. name: default-slack-secret
  10. namespace: kubesphere-monitoring-system
  11. type: Opaque
  12. ---
  13. apiVersion: notification.kubesphere.io/v1alpha1
  14. kind: SlackConfig
  15. metadata:
  16. name: default-slack-config
  17. namespace: kubesphere-monitoring-system
  18. labels:
  19. app: notification-manager
  20. type: default
  21. spec:
  22. slackTokenSecret:
  23. key: token
  24. name: default-slack-secret
  25. EOF

Slack receivers with label type: tenant can only receive notifications from the namespace to which the specified tenant user has access. If you want them to receive notifications from all namespaces or even without a namespace label, you can create a global slack receiver with label type: global as below:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: notification.kubesphere.io/v1alpha1
  3. kind: SlackReceiver
  4. metadata:
  5. name: global-slack-slack
  6. namespace: kubesphere-monitoring-system
  7. labels:
  8. app: notification-manager
  9. type: global
  10. spec:
  11. channel: global
  12. EOF

Note

Global slack receiver will use the default slack config.