Bringing your own Prometheus

Even though the linkerd-viz extension comes with its own Prometheus instance, there can be cases where using an external instance makes more sense for various reasons.

This tutorial shows how to configure an external Prometheus instance to scrape both the control plane as well as the proxy’s metrics in a format that is consumable both by a user as well as Linkerd control plane components like web, etc.

There are two important points to tackle here.

  • Configuring external Prometheus instance to get the Linkerd metrics.
  • Configuring the linkerd-viz extension to use that Prometheus.

Prometheus Scrape Configuration

The following scrape configuration has to be applied to the external Prometheus instance.

Note

The below scrape configuration is a subset of the full linkerd-prometheus scrape configuration.

Before applying, it is important to replace templated values (present in {{}}) with direct values for the below configuration to work.

  1.     - job_name: 'linkerd-controller'
  2.       kubernetes_sd_configs:
  3.       - role: pod
  4.         namespaces:
  5.           names:
  6.           - '{{.Values.linkerdNamespace}}'
  7.           - '{{.Values.namespace}}'
  8.       relabel_configs:
  9.       - source_labels:
  10.         - __meta_kubernetes_pod_container_port_name
  11.         action: keep
  12.         regex: admin-http
  13.       - source_labels: [__meta_kubernetes_pod_container_name]
  14.         action: replace
  15.         target_label: component
  17.     - job_name: 'linkerd-service-mirror'
  18.       kubernetes_sd_configs:
  19.       - role: pod
  20.       relabel_configs:
  21.       - source_labels:
  22.         - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
  23.         - __meta_kubernetes_pod_container_port_name
  24.         action: keep
  25.         regex: linkerd-service-mirror;admin-http$
  26.       - source_labels: [__meta_kubernetes_pod_container_name]
  27.         action: replace
  28.         target_label: component
  30.     - job_name: 'linkerd-proxy'
  31.       kubernetes_sd_configs:
  32.       - role: pod
  33.       relabel_configs:
  34.       - source_labels:
  35.         - __meta_kubernetes_pod_container_name
  36.         - __meta_kubernetes_pod_container_port_name
  37.         - __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
  38.         action: keep
  39.         regex: ^{{default .Values.proxyContainerName "linkerd-proxy" .Values.proxyContainerName}};linkerd-admin;{{.Values.linkerdNamespace}}$
  40.       - source_labels: [__meta_kubernetes_namespace]
  41.         action: replace
  42.         target_label: namespace
  43.       - source_labels: [__meta_kubernetes_pod_name]
  44.         action: replace
  45.         target_label: pod
  46.       # special case k8s' "job" label, to not interfere with prometheus' "job"
  47.       # label
  48.       # __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =>
  49.       # k8s_job=foo
  50.       - source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
  51.         action: replace
  52.         target_label: k8s_job
  53.       # drop __meta_kubernetes_pod_label_linkerd_io_proxy_job
  54.       - action: labeldrop
  55.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
  56.       # __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =>
  57.       # deployment=foo
  58.       - action: labelmap
  59.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
  60.       # drop all labels that we just made copies of in the previous labelmap
  61.       - action: labeldrop
  62.         regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
  63.       # __meta_kubernetes_pod_label_linkerd_io_foo=bar =>
  64.       # foo=bar
  65.       - action: labelmap
  66.         regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
  67.       # Copy all pod labels to tmp labels
  68.       - action: labelmap
  69.         regex: __meta_kubernetes_pod_label_(.+)
  70.         replacement: __tmp_pod_label_$1
  71.       # Take `linkerd_io_` prefixed labels and copy them without the prefix
  72.       - action: labelmap
  73.         regex: __tmp_pod_label_linkerd_io_(.+)
  74.         replacement:  __tmp_pod_label_$1
  75.       # Drop the `linkerd_io_` originals
  76.       - action: labeldrop
  77.         regex: __tmp_pod_label_linkerd_io_(.+)
  78.       # Copy tmp labels into real labels
  79.       - action: labelmap
  80.         regex: __tmp_pod_label_(.+)

The running configuration of the builtin prometheus can be used as a reference.

  1. kubectl -n linkerd-viz  get configmap prometheus-config -o yaml

Linkerd-Viz Extension Configuration

Linkerd’s viz extension components like metrics-api, etc depend on the Prometheus instance to power the dashboard and CLI.

The prometheusUrl field gives you a single place through which all these components can be configured to an external Prometheus URL. This is allowed both through the CLI and Helm.

CLI

This can be done by passing a file with the above field to the values flag, which is available through linkerd viz install command.

  1. prometheusUrl: existing-prometheus.xyz:9090

Once applied, this configuration is not persistent across installs. The same has to be passed again by the user during re-installs, upgrades, etc.

When using an external Prometheus and configuring the prometheusUrl field, Linkerd’s Prometheus will still be included in installation. If you wish to disable it, be sure to include the following configuration as well:

  1. prometheus:
  2.   enabled: false

Helm

The same configuration can be applied through values.yaml when using Helm. Once applied, Helm makes sure that the configuration is persistent across upgrades.

More information on installation through Helm can be found here