Debugging Virtual Machines

This page describes how to troubleshoot issues with Istio deployed to Virtual Machines. Before reading this, you should take the steps in Virtual Machine Installation. Additionally, Virtual Machine Architecture can help you understand how the components interact.

Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.

While much of the same information is available on both platforms, accessing this information differs.

Monitoring health

The Istio sidecar is typically run as a systemd unit. To ensure its running properly, you can check that status:

  1. $ systemctl status istio

Additionally, the sidecar health can be programmatically check at its health endpoint:

  1. $ curl localhost:15021/healthz/ready -I

Logs

Logs for the Istio proxy can be found in a few places.

To access the systemd logs, which has details about the initialization of the proxy:

  1. $ journalctl -f -u istio -n 1000

The proxy will redirect stderr and stdout to /var/log/istio/istio.err.log and /var/log/istio/istio.log, respectively. To view these in a format similar to kubectl:

  1. $ tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100

Log levels can be modified by changing the cluster.env configuration file. Make sure to restart istio if it is already running:

  1. $ echo "ISTIO_AGENT_FLAGS=\"--log_output_level=dns:debug --proxyLogLevel=debug\"" >> /var/lib/istio/envoy/cluster.env
  2. $ systemctl restart istio

Iptables

To ensure iptables rules have been successfully applied:

  1. $ sudo iptables-save
  2. ...
  3. -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
  4. -A ISTIO_OUTPUT -j ISTIO_REDIRECT

Istioctl

Most istioctl commands will function properly with virtual machines. For example, istioctl proxy-status can be used to view all connected proxies:

  1. $ istioctl proxy-status
  2. NAME CDS LDS EDS RDS ISTIOD VERSION
  3. vm-1.default SYNCED SYNCED SYNCED SYNCED istiod-789ffff8-f2fkt 1.15.0

However, istioctl proxy-config relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines. Instead, a file containing the configuration dump from Envoy can be passed. For example:

  1. $ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
  2. SERVICE FQDN PORT SUBSET DIRECTION TYPE
  3. istiod.istio-system.svc.cluster.local 443 - outbound EDS
  4. istiod.istio-system.svc.cluster.local 15010 - outbound EDS
  5. istiod.istio-system.svc.cluster.local 15012 - outbound EDS
  6. istiod.istio-system.svc.cluster.local 15014 - outbound EDS

Automatic registration

When a virtual machine connects to Istiod, a WorkloadEntry will automatically be created. This enables the virtual machine to become a part of a Service, similar to an Endpoint in Kubernetes.

To check these are created correctly:

  1. $ kubectl get workloadentries
  2. NAME AGE ADDRESS
  3. vm-10.128.0.50 14m 10.128.0.50

Certificates

Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates.

The status of these certificates can be viewed in the same way as in Kubernetes:

  1. $ curl -s localhost:15000/config_dump | ./istioctl proxy-config secret --file -
  2. RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
  3. default Cert Chain ACTIVE true 251932493344649542420616421203546836446 2021-01-29T18:07:21Z 2021-01-28T18:07:21Z
  4. ROOTCA CA ACTIVE true 81663936513052336343895977765039160718 2031-01-26T17:54:44Z 2021-01-28T17:54:44Z

Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.

  1. $ ls /etc/certs
  2. cert-chain.pem key.pem root-cert.pem