How-To: Observe metrics with Grafana

How to view Dapr metrics in a Grafana dashboard.

Available dashboards

The grafana-system-services-dashboard.json template shows Dapr system component status, dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement:

Screenshot of the system service dashboard

The grafana-sidecar-dashboard.json template shows Dapr sidecar status, including sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.:

Screenshot of the sidecar dashboard

The grafana-actor-dashboard.json template shows Dapr Sidecar status, actor invocation throughput/latency, timer/reminder triggers, and turn-based concurrnecy:

Screenshot of the actor dashboard

Pre-requisites

Setup on Kubernetes

Install Grafana

  1. Add the Grafana Helm repo:

    1. helm repo add grafana https://grafana.github.io/helm-charts
  2. Install the chart:

    1. helm install grafana grafana/grafana -n dapr-monitoring

    Note

    If you are Minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command instead:

    1. helm install grafana grafana/grafana -n dapr-monitoring --set persistence.enabled=false
  3. Retrieve the admin password for Grafana login:

    1. kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

    You will get a password similar to cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%. Remove the % character from the password to get cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1 as the admin password.

  4. Validation Grafana is running in your cluster:

    1. kubectl get pods -n dapr-monitoring
    2. NAME READY STATUS RESTARTS AGE
    3. dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
    4. dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
    5. dapr-prom-prometheus-node-exporter-5jh8p 1/1 Running 0 4m58s
    6. dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 4m58s
    7. dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s
    8. dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s
    9. dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
    10. grafana-c49889cff-x56vj 1/1 Running 0 5m10s

Configure Prometheus as data source

First you need to connect Prometheus as a data source to Grafana.

  1. Port-forward to svc/grafana:

    1. kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring
    2. Forwarding from 127.0.0.1:8080 -> 3000
    3. Forwarding from [::1]:8080 -> 3000
    4. Handling connection for 8080
    5. Handling connection for 8080
  2. Open a browser to http://localhost:8080

  3. Login to Grafana

    • Username = admin
    • Password = Password from above
  4. Select Configuration and Data Sources

    Screenshot of the Grafana add Data Source menu

  5. Add Prometheus as a data source.

    Screenshot of the Prometheus add Data Source

  6. Get your Prometheus HTTP URL

    The Prometheus HTTP URL follows the format http://<prometheus service endpoint>.<namespace>

    Start by getting the Prometheus server endpoint by running the following command:

    1. kubectl get svc -n dapr-monitoring
    2. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    3. dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 <none> 8080/TCP 7d9h
    4. dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 <none> 80/TCP 7d9h
    5. dapr-prom-prometheus-node-exporter ClusterIP None <none> 9100/TCP 7d9h
    6. dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 <none> 9091/TCP 7d9h
    7. dapr-prom-prometheus-server ClusterIP 10.0.172.191 <none> 80/TCP 7d9h
    8. elasticsearch-master ClusterIP 10.0.36.146 <none> 9200/TCP,9300/TCP 7d10h
    9. elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 7d10h
    10. grafana ClusterIP 10.0.15.229 <none> 80/TCP 5d5h
    11. kibana-kibana ClusterIP 10.0.188.224 <none> 5601/TCP 7d10h

    In this guide the server name is dapr-prom-prometheus-server and the namespace is dapr-monitoring, so the HTTP URL will be http://dapr-prom-prometheus-server.dapr-monitoring.

  7. Fill in the following settings:

    • Name: Dapr
    • HTTP URL: http://dapr-prom-prometheus-server.dapr-monitoring
    • Default: On

    Screenshot of the Prometheus Data Source configuration

  8. Click Save & Test button to verify that the connection succeeded.

Import dashboards in Grafana

  1. In the upper left corner of the Grafana home screen, click the “+” option, then “Import”.

    You can now import Grafana dashboard templates from release assets for your Dapr version:

    Screenshot of the Grafana dashboard upload option

  2. Find the dashboard that you imported and enjoy

    Screenshot of Dapr service dashboard

    Tip

    Hover your mouse over the i in the corner to the description of each chart:

    Screenshot of the tooltip for graphs

References

Example

Last modified September 17, 2021 : Merge pull request #1757 from georgestevens99/1440SecretKeyRefExplanation (620a5f8)