Prometheus Integration

Purpose

Prometheus server can monitor various metrics and provide an observation of the Antrea Controller and Agent components. The doc provides general guidelines to the configuration of Prometheus server to operate with the Antrea components.

About Prometheus

Prometheus is an open source monitoring and alerting server. Prometheus is capable of collecting metrics from various Kubernetes components, storing and providing alerts. Prometheus can provide visibility by integrating with other products such as Grafana.

One of Prometheus capabilities is self-discovery of Kubernetes services which expose their metrics. So Prometheus can scrape the metrics of any additional components which are added to the cluster without further configuration changes.

Antrea Configuration

Enable Prometheus metrics listener by setting enablePrometheusMetrics parameter to true in the Controller and the Agent configurations.

Prometheus Configuration

Prometheus RBAC

Prometheus requires access to Kubernetes API resources for the service discovery capability. Reading metrics also requires access to the “/metrics” API endpoints.

  1. apiVersion: rbac.authorization.k8s.io/v1beta1
  2. kind: ClusterRole
  3. metadata:
  4.   name: prometheus
  5. rules:
  6. - apiGroups: [""]
  7.   resources:
  8.   - nodes
  9.   - nodes/proxy
  10.   - services
  11.   - endpoints
  12.   - pods
  13.   verbs: ["get", "list", "watch"]
  14. - apiGroups:
  15.   - networking.k8s.io
  16.   resources:
  17.   - ingresses
  18.   verbs: ["get", "list", "watch"]
  19. - nonResourceURLs: ["/metrics"]
  20.   verbs: ["get"]

Antrea Metrics Listener Access

To scrape the metrics from Antrea Controller and Agent, Prometheus needs the following permissions

  1. kind: ClusterRole
  2. apiVersion: rbac.authorization.k8s.io/v1beta1
  3. metadata:
  4.   name: prometheus-antrea
  5. rules:
  6. - nonResourceURLs:
  7.   - /metrics
  8.   verbs:
  9.   - get

Antrea Components Scraping configuration

Add the following jobs to Prometheus scraping configuration to enable metrics collection from Antrea components

Controller Scraping

  1. - job_name: 'antrea-controllers'
  2. kubernetes_sd_configs:
  3. - role: endpoints
  4. scheme: https
  5. tls_config:
  6.   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  7.   insecure_skip_verify: true
  8. bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  9. relabel_configs:
  10. - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_container_name]
  11.   action: keep
  12.   regex: kube-system;antrea-controller

Agent Scraping

  1. - job_name: 'antrea-agents'
  2. kubernetes_sd_configs:
  3. - role: pod
  4. scheme: https
  5. tls_config:
  6.   ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  7.   insecure_skip_verify: true
  8. bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  9. relabel_configs:
  10. - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_container_name]
  11.   action: keep
  12.   regex: kube-system;antrea-agent

For further reference see the enclosed configuration file.

The configuration file above can be used to deploy Prometheus Server with scraping configuration for Antrea services. To deploy this configuration use kubectl apply -f build/yamls/antrea-prometheus.yml