Customizing Istio Metrics

This task shows you how to customize the metrics that Istio generates.

Istio generates telemetry that various dashboards consume to help you visualize your mesh. For example, dashboards that support Istio include:

By default, Istio defines and generates a set of standard metrics (e.g. requests_total), but you can also customize them and create new metrics.

Custom statistics configuration

Istio uses the Envoy proxy to generate metrics and provides its configuration in the EnvoyFilter at manifests/charts/istio-control/istio-discovery/templates/telemetryv2_1.7.yaml.

Configuring custom statistics involves two sections of the EnvoyFilter: definitions and metrics. The definitions section supports creating new metrics by name, the expected value expression, and the metric type (counter, gauge, and histogram). The metrics section provides values for the metric dimensions as expressions, and allows you to remove or override the existing metric dimensions. You can modify the standard metric definitions using tags_to_remove or by re-defining a dimension. These configuration settings are also exposed as istioctl installation options, which allow you to customize different metrics for gateways and sidecars as well as for the inbound or outbound direction.

For more information, see Stats Config reference.

Before you begin

Install Istio in your cluster and deploy an application. Alternatively, you can set up custom statistics as part of the Istio installation.

The Bookinfo sample application is used as the example application throughout this task.

Enable custom metrics

  1. The default telemetry v2 EnvoyFilter configuration is equivalent to the following installation options:

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4.   values:
    5.     telemetry:
    6.       v2:
    7.         prometheus:
    8.           configOverride:
    9.             inboundSidecar:
    10.               disable_host_header_fallback: false
    11.             outboundSidecar:
    12.               disable_host_header_fallback: false
    13.             gateway:
    14.               disable_host_header_fallback: true

    To customize telemetry v2 metrics, for example, to add request_host and destination_port dimensions to the requests_total metric emitted by both gateways and sidecars in the inbound and outbound direction, change the installation options as follows:

    You only need to specify the configuration for the settings that you want to customize. For example, to only customize the sidecar inbound requests_count metric, you can omit the outboundSidecar and gateway sections in the configuration. Unspecified settings will retain the default configuration, equivalent to the explicit settings shown above.

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4.   values:
    5.     telemetry:
    6.       v2:
    7.         prometheus:
    8.           configOverride:
    9.             inboundSidecar:
    10.               metrics:
    11.                 - name: requests_total
    12.                   dimensions:
    13.                     destination_port: string(destination.port)
    14.                     request_host: request.host
    15.             outboundSidecar:
    16.               metrics:
    17.                 - name: requests_total
    18.                   dimensions:
    19.                     destination_port: string(destination.port)
    20.                     request_host: request.host
    21.             gateway:
    22.               metrics:
    23.                 - name: requests_total
    24.                   dimensions:
    25.                     destination_port: string(destination.port)
    26.                     request_host: request.host

  2. Apply the following annotation to all injected pods with the list of the dimensions to extract into a Prometheus time series using the following command:

    This step is needed only if your dimensions are not already in DefaultStatTags list

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. spec:
    4.   template: # pod template
    5.     metadata:
    6.       annotations:
    7.         sidecar.istio.io/extraStatTags: destination_port,request_host

    To enable extra tags mesh wide, you can add extraStatTags to your mesh config:

    1. meshConfig:
    2.   defaultConfig:
    3.     extraStatTags:
    4.      - destination_port
    5.      - request_host

Verify the results

Send traffic to the mesh. For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:

  1. $ curl "http://$GATEWAY_URL/productpage"

$GATEWAY_URL is the value set in the Bookinfo example.

Use the following command to verify that Istio generates the data for your new or modified dimensions:

  1. $ kubectl exec "$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_requests_total

For example, in the output, locate the metric istio_requests_total and verify it contains your new dimension.

It might take a short period of time for the proxies to start applying the config. If the metric is not received, you may retry sending requests after a short wait, and look for the metric again.

Use expressions for values

The values in the metric configuration are common expressions, which means you must double-quote strings in JSON, e.g. “‘string value’”. Unlike Mixer expression language, there is no support for the pipe (|) operator, but you can emulate it with the has or in operator, for example:

  1. has(request.host) ? request.host : "unknown"

For more information, see Common Expression Language.

Istio exposes all standard Envoy attributes. Peer metadata is available as attributes upstream_peer for outbound and downstream_peer for inbound with the following fields:

FieldTypeValue
namestringName of the pod.
namespacestringNamespace that the pod runs in.
labelsmapWorkload labels.
ownerstringWorkload owner.
workload_namestringWorkload name.
platform_metadatamapPlatform metadata with prefixed keys.
istio_versionstringVersion identifier for the proxy.
mesh_idstringUnique identifier for the mesh.
app_containerslist<string>List of short names for application containers.
cluster_idstringIdentifier for the cluster to which this workload belongs.

For example, the expression for the peer app label to be used in an outbound configuration is upstream_peer.labels['app'].value.

For more information, see configuration reference.

See also

Classifying Metrics Based on Request or Response (Experimental)

This task shows you how to improve telemetry by grouping requests and responses by their type.

Collecting Metrics for TCP Services

This task shows you how to configure Istio to collect metrics for TCP services.

Querying Metrics from Prometheus

This task shows you how to query for Istio Metrics using Prometheus.

Reworking our Addon Integrations

A new way to manage installation of telemetry addons.

Mixer and the SPOF Myth

Improving availability and reducing latency.

Mixer Adapter Model

Provides an overview of Mixer’s plug-in architecture.