Inspecting Network Flows with the CLI

This guide walks you through using the Hubble CLI to inspect network flows and gain visibility into what is happening on the network level.

The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

Tip

This guide assumes that Cilium has been correctly installed in your Kubernetes cluster and that Hubble has been enabled. Please see Quick Installation and Setting up Hubble Observability for more information. If unsure, run cilium status and validate that Cilium and Hubble are up and running.

Note

This guide uses examples based on the Demo App. If you would like to run them, deploy the Demo App first. Please refer to Identity-Aware and HTTP-Aware Policy Enforcement for more details.

Inspecting the cluster’s network traffic with Hubble Relay

Let’s issue some requests to emulate some traffic again. This first request is allowed by the policy.

  1. kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  2. Ship landed

This next request is accessing an HTTP endpoint which is denied by policy.

  1. kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
  2. Access denied

Finally, this last request will hang because the xwing pod does not have the org=empire label required by policy. Press Control-C to kill the curl request, or wait for it to time out.

  1. kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
  2. command terminated with exit code 28

Let’s now inspect this traffic using the CLI. The command below filters all traffic on the application layer (L7, HTTP) to the deathstar pod:

  1. hubble observe --pod deathstar --protocol http
  2. May  4 13:23:40.501: default/tiefighter:42690 -> default/deathstar-c74d84667-cx5kp:80 http-request FORWARDED (HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing)
  3. May  4 13:23:40.502: default/tiefighter:42690 <- default/deathstar-c74d84667-cx5kp:80 http-response FORWARDED (HTTP/1.1 200 0ms (POST http://deathstar.default.svc.cluster.local/v1/request-landing))
  4. May  4 13:23:43.791: default/tiefighter:42742 -> default/deathstar-c74d84667-cx5kp:80 http-request DROPPED (HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port)

The following command shows all traffic to the deathstar pod that has been dropped:

  1. hubble observe --pod deathstar --verdict DROPPED
  2. May  4 13:23:43.791: default/tiefighter:42742 -> default/deathstar-c74d84667-cx5kp:80 http-request DROPPED (HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port)
  3. May  4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
  4. May  4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
  5. May  4 13:23:48.854: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)

Feel free to further inspect the traffic. To get help for the observe command, use hubble help observe.