OpenTelemetry

The Envoy proxies can be configured to export their access logs in OpenTelemetry format. In this example, the proxies send access logs to an OpenTelemetry collector, which is configured to print the logs to standard output. The standard output of the OpenTelemetry collector can then be accessed via the kubectl logs command.

Before you begin

  • Setup Istio by following the instructions in the Installation guide.

    The egress gateway and access logging will be enabled if you install the demo configuration profile.

  • Deploy the sleep sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:

    Zip

    1. $ kubectl apply -f @samples/sleep/sleep.yaml@

    Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

    You can use any pod with curl installed as a test source.

  • Set the SOURCE_POD environment variable to the name of your source pod:

    1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
  • Start the httpbin sample.

    If you have enabled automatic sidecar injection, deploy the httpbin service:

    Zip

    1. $ kubectl apply -f @samples/httpbin/httpbin.yaml@

    Otherwise, you have to manually inject the sidecar before deploying the httpbin application:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@)
  • Start the otel-collector sample.

    Zip

    1. $ kubectl apply -f @samples/open-telemetry/otel.yaml@ -n istio-system

Enable Envoy’s access logging

To enable access logging, use the Telemetry API.

Edit MeshConfig to add an OpenTelemetry provider, named otel. This involves adding an extension provider stanza:

  1. extensionProviders:
  2. - name: otel
  3. envoyOtelAls:
  4. service: opentelemetry-collector.istio-system.svc.cluster.local
  5. port: 4317

The final configuration should look something like:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: istio
  5. namespace: istio-system
  6. data:
  7. mesh: |-
  8. accessLogFile: /dev/stdout
  9. defaultConfig:
  10. discoveryAddress: istiod.istio-system.svc:15012
  11. proxyMetadata: {}
  12. tracing:
  13. zipkin:
  14. address: zipkin.istio-system:9411
  15. enablePrometheusMerge: true
  16. extensionProviders:
  17. - name: otel
  18. envoyOtelAls:
  19. service: opentelemetry-collector.istio-system.svc.cluster.local
  20. port: 4317
  21. rootNamespace: istio-system
  22. trustDomain: cluster.local
  23. meshNetworks: 'networks: {}'

Next, add a Telemetry resource that tells Istio to send access logs to the OpenTelemetry collector.

  1. $ cat <<EOF | kubectl apply -n default -f -
  2. apiVersion: telemetry.istio.io/v1alpha1
  3. kind: Telemetry
  4. metadata:
  5. name: sleep-logging
  6. spec:
  7. selector:
  8. matchLabels:
  9. app: sleep
  10. accessLogging:
  11. - providers:
  12. - name: otel
  13. EOF

The above example uses the otel access log provider, and we do not configure anything other than default settings.

Similar configuration can also be applied on an individual namespace, or to an individual workload, to control logging at a fine grained level.

For more information about using the Telemetry API, see the Telemetry API overview.

Using Mesh Config

If you used an IstioOperator CR to install Istio, add the following field to your configuration:

  1. spec:
  2. meshConfig:
  3. accessLogFile: /dev/stdout
  4. extensionProviders:
  5. - name: otel
  6. envoyOtelAls:
  7. service: opentelemetry-collector.istio-system.svc.cluster.local
  8. port: 4317
  9. defaultProviders:
  10. accessLogging:
  11. - envoy
  12. - otel

Otherwise, add the equivalent setting to your original istioctl install command, for example:

  1. $ istioctl install -f <your-istio-operator-config-file>

Default access log format

Istio will use the following default access log format if accessLogFormat is not specified:

  1. [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS%
  2. \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\"
  3. \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n

The following table shows an example using the default access log format for a request sent from sleep to httpbin:

Log operatoraccess log in sleepaccess log in httpbin
[%STARTTIME%][2020-11-25T21:26:18.409Z][2020-11-25T21:26:18.409Z]
\”%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\”“GET /status/418 HTTP/1.1”“GET /status/418 HTTP/1.1”
%RESPONSE_CODE%418418
%RESPONSE_FLAGS%--
%RESPONSE_CODE_DETAILS%via_upstreamvia_upstream
%CONNECTION_TERMINATION_DETAILS%--
\”%UPSTREAM_TRANSPORT_FAILURE_REASON%\”“-““-“
%BYTES_RECEIVED%00
%BYTES_SENT%135135
%DURATION%43
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%41
\”%REQ(X-FORWARDED-FOR)%\”“-““-“
\”%REQ(USER-AGENT)%\”“curl/7.73.0-DEV”“curl/7.73.0-DEV”
\”%REQ(X-REQUEST-ID)%\”“84961386-6d84-929d-98bd-c5aee93b5c88”“84961386-6d84-929d-98bd-c5aee93b5c88”
\”%REQ(:AUTHORITY)%\”“httpbin:8000”“httpbin:8000”
\”%UPSTREAM_HOST%\”“10.44.1.27:80”“127.0.0.1:80”
%UPSTREAM_CLUSTER%outbound|8000||httpbin.foo.svc.cluster.localinbound|8000||
%UPSTREAM_LOCAL_ADDRESS%10.44.1.23:37652127.0.0.1:41854
%DOWNSTREAM_LOCAL_ADDRESS%10.0.45.184:800010.44.1.27:80
%DOWNSTREAM_REMOTE_ADDRESS%10.44.1.23:4652010.44.1.23:37652
%REQUESTED_SERVER_NAME%-outbound.8000..httpbin.foo.svc.cluster.local
%ROUTE_NAME%defaultdefault

Test the access log

  1. Send a request from sleep to httpbin:

    1. $ kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -v httpbin:8000/status/418
    2. ...
    3. < HTTP/1.1 418 Unknown
    4. < server: envoy
    5. ...
    6. -=[ teapot ]=-
    7. _...._
    8. .' _ _ `.
    9. | ."` ^ `". _,
    10. \_;`"---"`|//
    11. | ;/
    12. \_ _/
    13. `"""`
  2. Check otel-collector’s log:

    1. $ kubectl logs -l app=opentelemetry-collector -n istio-system
    2. [2020-11-25T21:26:18.409Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 1 "-" "curl/7.73.0-DEV" "84961386-6d84-929d-98bd-c5aee93b5c88" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:41854 10.44.1.27:80 10.44.1.23:37652 outbound_.8000_._.httpbin.foo.svc.cluster.local default

Note that the messages corresponding to the request appear in logs of the Istio proxies of both the source and the destination, sleep and httpbin, respectively. You can see in the log the HTTP verb (GET), the HTTP path (/status/418), the response code (418) and other request-related information.

Cleanup

Shutdown the sleep and httpbin services:

ZipZipZip

  1. $ kubectl delete telemetry sleep-logging
  2. $ kubectl delete -f @samples/sleep/sleep.yaml@
  3. $ kubectl delete -f @samples/httpbin/httpbin.yaml@
  4. $ kubectl delete -f @samples/open-telemetry/otel.yaml@

Disable Envoy’s access logging

Remove, or set to "", the meshConfig.extensionProviders and meshConfig.defaultProviders setting in your Istio install configuration.

In the example below, replace default with the name of the profile you used when you installed Istio.

  1. $ istioctl install --set profile=default
  2. Istio core installed
  3. Istiod installed
  4. Ingress gateways installed
  5. Installation complete