Traffic Metrics

Kuma facilitates consistent traffic metrics across all dataplanes in your mesh.

A user can enable traffic metrics by editing a Mesh resource and providing the desired Mesh-wide configuration. If necessary, metrics configuration can be customized for each Dataplane individually, e.g. to override the default metrics port that might be already in use on that particular machine.

Out-of-the-box, Kuma provides full integration with Prometheus:

  • if enabled, every dataplane will expose its metrics in Prometheus format
  • furthemore, Kuma will make sure that Prometheus can automatically find every dataplane in the mesh

On Universal

Enable Prometheus metrics per Mesh

To enable Prometheus metrics on every dataplane in the mesh, configure a Mesh resource as follows:

  1. type: Mesh
  2. name: default
  3. metrics:
  4.   prometheus: {}

which is a convenient shortcut for

  1. type: Mesh
  2. name: default
  3. metrics:
  4.   prometheus:
  5.     port: 5670
  6.     path: /metrics

Both snippets from above instruct Kuma to configure every dataplane in the mesh default to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

Override Prometheus settings per Dataplane

To override Mesh-wide defaults on a particular machine, configure Dataplane resource as follows:

  1. type: Dataplane
  2. mesh: default
  3. name: example
  4. metrics:
  5.   prometheus:
  6.     port: 1234
  7.     path: /non-standard-path

As a result, this particular dataplane will expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Configure dataplane discovery by Prometheus

Although dataplane metrics are now exposed, Prometheus doesn’t know anything about it just yet.

To help Prometheus to automatically discover dataplanes, Kuma provides a helper tool - kuma-prometheus-sd.

kuma-prometheus-sd is meant to run alongside Prometheus instance.

It knows where Kuma Control Plane is, it knows how to talk to it, it knows how to fetch an up-to-date list of dataplanes from it.

It then transforms that information into a format that Prometheus can understand, and saves it into a file on disk.

Prometheus watches for changes to that file and updates its scraping configuration accordingly.

First, you need to run kuma-prometheus-sd, e.g. by using the following command:

  1. kuma-prometheus-sd run \
  2.   --cp-address=http://kuma-control-plane.internal:5681 \
  3.   --output-file=/var/run/kuma-prometheus-sd/kuma.file_sd.json

The above configuration tells kuma-prometheus-sd to talk to Kuma Control Plane at http://kuma-control-plane.internal:5681Traffic Metrics - 图1 and save the list of dataplanes to /var/run/kuma-prometheus-sd/kuma.file_sd.json.

Then, you need to set up Prometheus to read from that file, e.g. by using prometheus.yml config with the following contents:

  1. scrape_configs:
  2. - job_name: 'kuma-dataplanes'
  3.   scrape_interval: 15s
  4.   file_sd_configs:
  5.   - files:
  6.     - /var/run/kuma-prometheus-sd/kuma.file_sd.json

and running

  1. prometheus --config.file=prometheus.yml

Now, if you check Targets page on Prometheus UI, you should see a list of dataplanes from your mesh, e.g.

A screenshot of Targets page on Prometheus UI

On Kubernetes

Enable Prometheus metrics per Mesh

To enable Prometheus metrics on every dataplane in the mesh, configure a Mesh resource as follows:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   metrics:
  7.     prometheus: {}

which is a convenient shortcut for

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   metrics:
  7.     prometheus:
  8.       port: 5670
  9.       path: /metrics

Both snippets from above instruct Kuma to configure every dataplane in the mesh default to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

Override Prometheus settings per Dataplane

To override Mesh-wide defaults for a particular Pod, use Kuma-specific annotations:

  • prometheus.metrics.kuma.io/port - to override Mesh-wide default port
  • prometheus.metrics.kuma.io/path - to override Mesh-wide default path

E.g.,

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   namespace: kuma-example
  5.   name: kuma-tcp-echo
  6. spec:
  7.   ...
  8.   template:
  9.     metadata:
  10.       ...
  11.       annotations:
  12.         prometheus.metrics.kuma.io/port: "1234"               # override Mesh-wide default port
  13.         prometheus.metrics.kuma.io/path: "/non-standard-path" # override Mesh-wide default path
  14.     spec:
  15.       containers:
  16.       ...

As a result, dataplane for this particular Pod will expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Configure dataplane discovery by Prometheus

On kubernetes, there are two ways of configuring dataplane discovery.

Annotations

Kuma will automatically annotate your Pod to be discovered by Prometheus, e.g.

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   namespace: kuma-example
  5.   name: kuma-tcp-echo
  6.   annotations:
  7.     prometheus.io/scrape: "true"   # will be automatically added by Kuma
  8.     prometheus.io/port: "5670"     # will be automatically added by Kuma
  9.     prometheus.io/path: "/metrics" # will be automatically added by Kuma
  10. spec:
  11.   containers:
  12.   ...

Notice usage of prometheus.io/* annotations to indicate where Prometheus should scrape metrics from.

Beware that Prometheus itself doesn’t have any knowledge about prometheus.io/* annotations.

Instead, they’re a part of configuration, which might differ from one Prometheus installation to another.

In particular, prometheus.io/* annotations are part of configuration used by Prometheus Helm chartTraffic Metrics - 图3.

If you’re using a different way to install Prometheus on kubernetes, those annotations might not have the desired effect.

Although it’s easy and works without any extra configuration, those annotations supports only one address to scrape. If you also have an application next to Kuma DP which exposes metrics through prometheus.io/* annotations, you have to use the other way to expose metrics for both the application and Kuma DP.

Kuma Prometheus SD

You can deploy kuma-prometheus-sd container next to you Prometheus instance just like in Universal setup.

First, add a volume to your Prometheus deployment to which kuma-prometheus-sd will write a file with the list of the dataplanes and from which Prometheus will read the list.

  1. volumes:
  2. - name: kuma-prometheus-sd-volume
  3.   emptyDir: {}

Then add a new container with kuma-prometheus-sd. It will connect to the control plane at given address and produce file to /var/run/kuma.io/prometheus-sd/kuma.file_sd.json in created volume.

  1.   containers:
  2. - name: kuma-prometheus-sd
  3.   image: kong-docker-kuma-docker.bintray.io/kuma-prometheus-sd:0.3.2
  4.   imagePullPolicy: Always
  5.   args:
  6.     - run
  7.     - --name=kuma-prometheus-sd
  8.     - --cp-address=http://kuma-control-plane.kuma-system:5681
  9.     - --output-file=/var/run/kuma.io/prometheus-sd/kuma.file_sd.json
  10.   volumeMounts:
  11.     - mountPath: "/var/run/kuma.io/prometheus-sd"
  12.       name: kuma-prometheus-sd-volume

Next step is to mount the volume to the Prometheus container

  1. volumeMounts:
  2. - mountPath: "/var/run/kuma.io/prometheus-sd"
  3.   name: kuma-prometheus-sd-volume

Finally, modify your Prometheus config to use generated file

  1. - job_name: 'kuma-dataplanes'
  2.   scrape_interval: "5s"
  3.   file_sd_configs:
  4.   - files:
  5.     - /var/run/kuma.io/prometheus-sd/kuma.file_sd.json

Refer to full example of the deployment and the configuration.

In this way of integrating Kuma with Prometheus, you can still use prometheus.io/* for your applications.