Configurability (Beta/Development)

Istio provides the ability to configure advanced tracing options, including sampling rates and span tags. Sampling is a beta feature, but custom tags and tracing tag length are considered in-development for this release.

Create a MeshConfig with trace settings

All tracing options are configured by using MeshConfig during Istio installation. To simplify configuration, create a single YAML file to pass to istioctl.

  1. cat <<'EOF' > tracing.yaml
  2. apiVersion: install.istio.io/v1alpha1
  3. kind: IstioOperator
  4. spec:
  5.   meshConfig:
  6.     defaultConfig:
  7.       tracing:
  8. EOF

Then, you can append any configuration options to the tracing.yaml file.

Trace sampling

Istio captures a trace for all requests by default when installing with the demo profile. For example, when using the Bookinfo sample application, every time you access /productpage you see a corresponding trace in the dashboard. This sampling rate is suitable for a test or low traffic mesh. For a high traffic mesh you can lower the trace sampling percentage in one of two ways:

Previously, the recommended method was to change the values.pilot.traceSampling setting during the mesh setup or to change the PILOT_TRACE_SAMPLE environment variable in the pilot or istiod deployment. While this method to alter sampling continues to work, the following method is strongly recommended instead.

In the event that both are specified, the value specified in the MeshConfig will override any other setting.

To modify the default random sampling, which is defaulted to a value of 100 in the demo profile and 1 for the default profile, add the following to your tracing.yaml file.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         sampling: <VALUE>

Where the <VALUE> should be in the range of 0.0 to 100.0 with a precision of 0.01. For example, to trace 5 requests out of every 10000, use 0.05 as the value here.

Customizing tracing tags

The ability to add custom tracing tags to spans has also been implemented.

Tags can be added to spans based on literals, environmental variables and client request headers.

There is no limit on the number of custom tags that you can add, but tag names must be unique.

To add custom tags to your spans, add the following to your tracing.yaml file.

Literals:

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         custom_tags:
  8.           tag_literal:                   # user-defined name
  9.             literal:
  10.               value: <VALUE>

Environmental variables:

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         custom_tags:
  8.           tag_env:                       # user-defined name
  9.             environment:
  10.               name: <ENV_VARIABLE_NAME>
  11.               defaultValue: <VALUE>      # optional

In order to add custom tags based on environmental variables, you must modify the istio-sidecar-injector ConfigMap in your root Istio system namespace.

Client request headers:

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         custom_tags:
  8.           tag_header:                    # user-defined name
  9.             header:
  10.               name: <CLIENT-HEADER>
  11.               defaultValue: <VALUE>      # optional

Customizing tracing tag length

By default, the maximum length for the request path included as part of the HttpUrl span tag is 256.

To modify this maximum length, add the following to your tracing.yaml file.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       tracing:
  7.         max_path_tag_length: <VALUE>

See also

Jaeger

Learn how to configure the proxies to send tracing requests to Jaeger.

Lightstep

How to configure the proxies to send tracing requests to Lightstep.

Overview

Overview of distributed tracing in Istio.

Remotely Accessing Telemetry Addons

This task shows you how to configure external access to the set of Istio telemetry addons.

Zipkin

Learn how to configure the proxies to send tracing requests to Zipkin.

Mixer and the SPOF Myth

Improving availability and reducing latency.