Exporting Metrics

By design, Linkerd only keeps metrics data for a short, fixed window of time(currently, 6 hours). This means that if Linkerd’s metrics data is valuable toyou, you will probably want to export it into a full-fledged metrics store.

Internally, Linkerd stores its metrics in a Prometheus instance that runs aspart of the control plane. There are several basic approaches to exportingmetrics data from Linkerd:

Using the Prometheus federation API

If you are using Prometheus as your own metrics store, we recommend takingadvantage of Prometheus’s federation API, which is designed exactly for theuse case of copying data from one Prometheus to another.

Simply add the following item to your scrape_configs in your Prometheusconfig file (replace {{.Namespace}} with the namespace where Linkerd isrunning):

  1. - job_name: 'linkerd'
  2.   kubernetes_sd_configs:
  3.   - role: pod
  4.     namespaces:
  5.       names: ['{{.Namespace}}']
  7.   relabel_configs:
  8.   - source_labels:
  9.     - __meta_kubernetes_pod_container_name
  10.     action: keep
  11.     regex: ^prometheus$
  13.   honor_labels: true
  14.   metrics_path: '/federate'
  16.   params:
  17.     'match[]':
  18.       - '{job="linkerd-proxy"}'
  19.       - '{job="linkerd-controller"}'

Alternatively, if you prefer to use Prometheus’ ServiceMonitors to configureyour Prometheus, you can use this ServiceMonitor YAML (replace {{.Namespace}}with the namespace where Linkerd is running):

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   labels:
  5.     k8s-app: linkerd-prometheus
  6.     release: monitoring
  7.   name: linkerd-federate
  8.   namespace: {{.Namespace}}
  9. spec:
  10.   endpoints:
  11.   - interval: 30s
  12.     scrapeTimeout: 30s
  13.     params:
  14.       match[]:
  15.       - '{job="linkerd-proxy"}'
  16.       - '{job="linkerd-controller"}'
  17.     path: /federate
  18.     port: admin-http
  19.     honorLabels: true
  20.     relabelings:
  21.     - action: keep
  22.       regex: '^prometheus$'
  23.       sourceLabels:
  24.       - '__meta_kubernetes_pod_container_name'
  25.   jobLabel: app
  26.   namespaceSelector:
  27.     matchNames:
  28.     - {{.Namespace}}
  29.   selector:
  30.     matchLabels:
  31.       linkerd.io/control-plane-component: prometheus

That’s it! Your Prometheus cluster is now configured to federate Linkerd’smetrics from Linkerd’s internal Prometheus instance.

Once the metrics are in your Prometheus, Linkerd’s proxy metrics will have thelabel job="linkerd-proxy" and Linkerd’s control plane metrics will have thelabel job="linkerd-controller". For more information on specific metric andlabel definitions, have a look at Proxy Metrics.

For more information on Prometheus’ /federate endpoint, have a look at thePrometheus federation docs.

Using a Prometheus integration

If you are not using Prometheus as your own long-term data store, you may beable to leverage one of Prometheus’s manyintegrations toautomatically extract data from Linkerd’s Prometheus instance into the datastore of your choice. Please refer to the Prometheus documentation for details.

Extracting data via Prometheus’s APIs

If neither Prometheus federation nor Prometheus integrations are options foryou, it is possible to call Prometheus’s APIs to extract data from Linkerd.

For example, you can call the federation API directly via a command like:

  1. curl -G \
  2.   --data-urlencode 'match[]={job="linkerd-proxy"}' \
  3.   --data-urlencode 'match[]={job="linkerd-controller"}' \
  4.   http://linkerd-prometheus.linkerd.svc.cluster.local:9090/federate

NoteIf your data store is outside the Kubernetes cluster, it is likely thatyou’ll want to set upingressat a domain name of your choice with authentication.

Similar to the /federate API, Prometheus provides a JSON query API toretrieve all metrics:

  1. curl http://linkerd-prometheus.linkerd.svc.cluster.local:9090/api/v1/query?query=request_total

Gathering data from the Linkerd proxies directly

Finally, if you want to avoid Linkerd’s Prometheus entirely, you can query theLinkerd proxies directly on their /metrics endpoint.

For example, to view /metrics from a single Linkerd proxy, running in thelinkerd namespace:

  1. kubectl -n linkerd port-forward \
  2.   $(kubectl -n linkerd get pods \
  3.     -l linkerd.io/control-plane-ns=linkerd \
  4.     -o jsonpath='{.items[0].metadata.name}') \
  5.   4191:4191

and then:

  1. curl localhost:4191/metrics