Remotely Accessing Telemetry Addons

This task shows how to configure Istio to expose and access the telemetry addons outside of a cluster.

Configuring remote access

Remote access to the telemetry addons can be configured in a number of different ways. This task covers two basic access methods: secure (via HTTPS) and insecure (via HTTP). The secure method is strongly recommended for any production or sensitive environment. Insecure access is simpler to set up, but will not protect any credentials or data transmitted outside of your cluster.

For both options, first follow these steps:

  1. Install Istio in your cluster.

    To additionally install the telemetry addons, follow the integrations documentation.

  2. Set up the domain to expose addons. In this example, you expose each addon on a subdomain, such as grafana.example.com.

    • If you have an existing domain pointing to the external IP address of istio-ingressgateway (say example.com):
    1. $ export INGRESS_DOMAIN="example.com"
    • If you do not have a domain, you may use nip.io which will automatically resolve to the IP address provided. This is not recommended for production usage.
    1. $ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
    2. $ export INGRESS_DOMAIN=${INGRESS_HOST}.nip.io

Option 1: Secure access (HTTPS)

A server certificate is required for secure access. Follow these steps to install and configure server certificates for a domain that you control.

This option covers securing the transport layer only. You should also configure the telemetry addons to require authentication when exposing them externally.

This example uses self-signed certificates, which may not be appropriate for production usages. For these cases, consider using cert-manager or other tools to provision certificates. You may also visit the Securing Gateways with HTTPS task for general information on using HTTPS on the gateway.

  1. Setup the certificates. This example uses openssl to self sign.

    1. $ CERT_DIR=/tmp/certs
    2. $ mkdir -p ${CERT_DIR}
    3. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj "/O=example Inc./CN=*.${INGRESS_DOMAIN}" -keyout ${CERT_DIR}/ca.key -out ${CERT_DIR}/ca.crt
    4. $ openssl req -out ${CERT_DIR}/cert.csr -newkey rsa:2048 -nodes -keyout ${CERT_DIR}/tls.key -subj "/CN=*.${INGRESS_DOMAIN}/O=example organization"
    5. $ openssl x509 -req -days 365 -CA ${CERT_DIR}/ca.crt -CAkey ${CERT_DIR}/ca.key -set_serial 0 -in ${CERT_DIR}/cert.csr -out ${CERT_DIR}/tls.crt
    6. $ kubectl create -n istio-system secret tls telemetry-gw-cert --key=${CERT_DIR}/tls.key --cert=${CERT_DIR}/tls.crt
  2. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: grafana-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 443
      13. name: https-grafana
      14. protocol: HTTPS
      15. tls:
      16. mode: SIMPLE
      17. credentialName: telemetry-gw-cert
      18. hosts:
      19. - "grafana.${INGRESS_DOMAIN}"
      20. ---
      21. apiVersion: networking.istio.io/v1alpha3
      22. kind: VirtualService
      23. metadata:
      24. name: grafana-vs
      25. namespace: istio-system
      26. spec:
      27. hosts:
      28. - "grafana.${INGRESS_DOMAIN}"
      29. gateways:
      30. - grafana-gateway
      31. http:
      32. - route:
      33. - destination:
      34. host: grafana
      35. port:
      36. number: 3000
      37. ---
      38. apiVersion: networking.istio.io/v1alpha3
      39. kind: DestinationRule
      40. metadata:
      41. name: grafana
      42. namespace: istio-system
      43. spec:
      44. host: grafana
      45. trafficPolicy:
      46. tls:
      47. mode: DISABLE
      48. ---
      49. EOF
      50. gateway.networking.istio.io/grafana-gateway created
      51. virtualservice.networking.istio.io/grafana-vs created
      52. destinationrule.networking.istio.io/grafana created
    2. Apply the following configuration to expose Kiali:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: kiali-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 443
      13. name: https-kiali
      14. protocol: HTTPS
      15. tls:
      16. mode: SIMPLE
      17. credentialName: telemetry-gw-cert
      18. hosts:
      19. - "kiali.${INGRESS_DOMAIN}"
      20. ---
      21. apiVersion: networking.istio.io/v1alpha3
      22. kind: VirtualService
      23. metadata:
      24. name: kiali-vs
      25. namespace: istio-system
      26. spec:
      27. hosts:
      28. - "kiali.${INGRESS_DOMAIN}"
      29. gateways:
      30. - kiali-gateway
      31. http:
      32. - route:
      33. - destination:
      34. host: kiali
      35. port:
      36. number: 20001
      37. ---
      38. apiVersion: networking.istio.io/v1alpha3
      39. kind: DestinationRule
      40. metadata:
      41. name: kiali
      42. namespace: istio-system
      43. spec:
      44. host: kiali
      45. trafficPolicy:
      46. tls:
      47. mode: DISABLE
      48. ---
      49. EOF
      50. gateway.networking.istio.io/kiali-gateway created
      51. virtualservice.networking.istio.io/kiali-vs created
      52. destinationrule.networking.istio.io/kiali created
    3. Apply the following configuration to expose Prometheus:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: prometheus-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 443
      13. name: https-prom
      14. protocol: HTTPS
      15. tls:
      16. mode: SIMPLE
      17. credentialName: telemetry-gw-cert
      18. hosts:
      19. - "prometheus.${INGRESS_DOMAIN}"
      20. ---
      21. apiVersion: networking.istio.io/v1alpha3
      22. kind: VirtualService
      23. metadata:
      24. name: prometheus-vs
      25. namespace: istio-system
      26. spec:
      27. hosts:
      28. - "prometheus.${INGRESS_DOMAIN}"
      29. gateways:
      30. - prometheus-gateway
      31. http:
      32. - route:
      33. - destination:
      34. host: prometheus
      35. port:
      36. number: 9090
      37. ---
      38. apiVersion: networking.istio.io/v1alpha3
      39. kind: DestinationRule
      40. metadata:
      41. name: prometheus
      42. namespace: istio-system
      43. spec:
      44. host: prometheus
      45. trafficPolicy:
      46. tls:
      47. mode: DISABLE
      48. ---
      49. EOF
      50. gateway.networking.istio.io/prometheus-gateway created
      51. virtualservice.networking.istio.io/prometheus-vs created
      52. destinationrule.networking.istio.io/prometheus created
    4. Apply the following configuration to expose the tracing service:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: tracing-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 443
      13. name: https-tracing
      14. protocol: HTTPS
      15. tls:
      16. mode: SIMPLE
      17. credentialName: telemetry-gw-cert
      18. hosts:
      19. - "tracing.${INGRESS_DOMAIN}"
      20. ---
      21. apiVersion: networking.istio.io/v1alpha3
      22. kind: VirtualService
      23. metadata:
      24. name: tracing-vs
      25. namespace: istio-system
      26. spec:
      27. hosts:
      28. - "tracing.${INGRESS_DOMAIN}"
      29. gateways:
      30. - tracing-gateway
      31. http:
      32. - route:
      33. - destination:
      34. host: tracing
      35. port:
      36. number: 80
      37. ---
      38. apiVersion: networking.istio.io/v1alpha3
      39. kind: DestinationRule
      40. metadata:
      41. name: tracing
      42. namespace: istio-system
      43. spec:
      44. host: tracing
      45. trafficPolicy:
      46. tls:
      47. mode: DISABLE
      48. ---
      49. EOF
      50. gateway.networking.istio.io/tracing-gateway created
      51. virtualservice.networking.istio.io/tracing-vs created
      52. destinationrule.networking.istio.io/tracing created
  3. Visit the telemetry addons via your browser.

    If you used self signed certificates, your browser will likely mark them as insecure.

    • Kiali: https://kiali.${INGRESS_DOMAIN}
    • Prometheus: https://prometheus.${INGRESS_DOMAIN}
    • Grafana: https://grafana.${INGRESS_DOMAIN}
    • Tracing: https://tracing.${INGRESS_DOMAIN}

Option 2: Insecure access (HTTP)

  1. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: grafana-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 80
      13. name: http-grafana
      14. protocol: HTTP
      15. hosts:
      16. - "grafana.${INGRESS_DOMAIN}"
      17. ---
      18. apiVersion: networking.istio.io/v1alpha3
      19. kind: VirtualService
      20. metadata:
      21. name: grafana-vs
      22. namespace: istio-system
      23. spec:
      24. hosts:
      25. - "grafana.${INGRESS_DOMAIN}"
      26. gateways:
      27. - grafana-gateway
      28. http:
      29. - route:
      30. - destination:
      31. host: grafana
      32. port:
      33. number: 3000
      34. ---
      35. apiVersion: networking.istio.io/v1alpha3
      36. kind: DestinationRule
      37. metadata:
      38. name: grafana
      39. namespace: istio-system
      40. spec:
      41. host: grafana
      42. trafficPolicy:
      43. tls:
      44. mode: DISABLE
      45. ---
      46. EOF
      47. gateway.networking.istio.io/grafana-gateway created
      48. virtualservice.networking.istio.io/grafana-vs created
      49. destinationrule.networking.istio.io/grafana created
    2. Apply the following configuration to expose Kiali:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: kiali-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 80
      13. name: http-kiali
      14. protocol: HTTP
      15. hosts:
      16. - "kiali.${INGRESS_DOMAIN}"
      17. ---
      18. apiVersion: networking.istio.io/v1alpha3
      19. kind: VirtualService
      20. metadata:
      21. name: kiali-vs
      22. namespace: istio-system
      23. spec:
      24. hosts:
      25. - "kiali.${INGRESS_DOMAIN}"
      26. gateways:
      27. - kiali-gateway
      28. http:
      29. - route:
      30. - destination:
      31. host: kiali
      32. port:
      33. number: 20001
      34. ---
      35. apiVersion: networking.istio.io/v1alpha3
      36. kind: DestinationRule
      37. metadata:
      38. name: kiali
      39. namespace: istio-system
      40. spec:
      41. host: kiali
      42. trafficPolicy:
      43. tls:
      44. mode: DISABLE
      45. ---
      46. EOF
      47. gateway.networking.istio.io/kiali-gateway created
      48. virtualservice.networking.istio.io/kiali-vs created
      49. destinationrule.networking.istio.io/kiali created
    3. Apply the following configuration to expose Prometheus:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: prometheus-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 80
      13. name: http-prom
      14. protocol: HTTP
      15. hosts:
      16. - "prometheus.${INGRESS_DOMAIN}"
      17. ---
      18. apiVersion: networking.istio.io/v1alpha3
      19. kind: VirtualService
      20. metadata:
      21. name: prometheus-vs
      22. namespace: istio-system
      23. spec:
      24. hosts:
      25. - "prometheus.${INGRESS_DOMAIN}"
      26. gateways:
      27. - prometheus-gateway
      28. http:
      29. - route:
      30. - destination:
      31. host: prometheus
      32. port:
      33. number: 9090
      34. ---
      35. apiVersion: networking.istio.io/v1alpha3
      36. kind: DestinationRule
      37. metadata:
      38. name: prometheus
      39. namespace: istio-system
      40. spec:
      41. host: prometheus
      42. trafficPolicy:
      43. tls:
      44. mode: DISABLE
      45. ---
      46. EOF
      47. gateway.networking.istio.io/prometheus-gateway created
      48. virtualservice.networking.istio.io/prometheus-vs created
      49. destinationrule.networking.istio.io/prometheus created
    4. Apply the following configuration to expose the tracing service:

      1. $ cat <<EOF | kubectl apply -f -
      2. apiVersion: networking.istio.io/v1alpha3
      3. kind: Gateway
      4. metadata:
      5. name: tracing-gateway
      6. namespace: istio-system
      7. spec:
      8. selector:
      9. istio: ingressgateway
      10. servers:
      11. - port:
      12. number: 80
      13. name: http-tracing
      14. protocol: HTTP
      15. hosts:
      16. - "tracing.${INGRESS_DOMAIN}"
      17. ---
      18. apiVersion: networking.istio.io/v1alpha3
      19. kind: VirtualService
      20. metadata:
      21. name: tracing-vs
      22. namespace: istio-system
      23. spec:
      24. hosts:
      25. - "tracing.${INGRESS_DOMAIN}"
      26. gateways:
      27. - tracing-gateway
      28. http:
      29. - route:
      30. - destination:
      31. host: tracing
      32. port:
      33. number: 80
      34. ---
      35. apiVersion: networking.istio.io/v1alpha3
      36. kind: DestinationRule
      37. metadata:
      38. name: tracing
      39. namespace: istio-system
      40. spec:
      41. host: tracing
      42. trafficPolicy:
      43. tls:
      44. mode: DISABLE
      45. ---
      46. EOF
      47. gateway.networking.istio.io/tracing-gateway created
      48. virtualservice.networking.istio.io/tracing-vs created
      49. destinationrule.networking.istio.io/tracing created
  2. Visit the telemetry addons via your browser.

    • Kiali: http://kiali.${INGRESS_DOMAIN}
    • Prometheus: http://prometheus.${INGRESS_DOMAIN}
    • Grafana: http://grafana.${INGRESS_DOMAIN}
    • Tracing: http://tracing.${INGRESS_DOMAIN}

Cleanup

  • Remove all related Gateways:

    1. $ kubectl -n istio-system delete gateway grafana-gateway kiali-gateway prometheus-gateway tracing-gateway
    2. gateway.networking.istio.io "grafana-gateway" deleted
    3. gateway.networking.istio.io "kiali-gateway" deleted
    4. gateway.networking.istio.io "prometheus-gateway" deleted
    5. gateway.networking.istio.io "tracing-gateway" deleted
  • Remove all related Virtual Services:

    1. $ kubectl -n istio-system delete virtualservice grafana-vs kiali-vs prometheus-vs tracing-vs
    2. virtualservice.networking.istio.io "grafana-vs" deleted
    3. virtualservice.networking.istio.io "kiali-vs" deleted
    4. virtualservice.networking.istio.io "prometheus-vs" deleted
    5. virtualservice.networking.istio.io "tracing-vs" deleted
  • Remove all related Destination Rules:

    1. $ kubectl -n istio-system delete destinationrule grafana kiali prometheus tracing
    2. destinationrule.networking.istio.io "grafana" deleted
    3. destinationrule.networking.istio.io "kiali" deleted
    4. destinationrule.networking.istio.io "prometheus" deleted
    5. destinationrule.networking.istio.io "tracing" deleted

See also

Reworking our Addon Integrations

A new way to manage installation of telemetry addons.

Jaeger

Learn how to configure the proxies to send tracing requests to Jaeger.

Zipkin

Learn how to configure the proxies to send tracing requests to Zipkin.

Configurability (Beta/Development)

How to configure tracing options (beta/development).

Jaeger

How to integrate with Jaeger.

Lightstep

How to configure the proxies to send tracing requests to Lightstep.