Tracing Openflow with ovnkube-trace

OVN and OVS traffic flows can be simulated in a single utility called ovnkube-trace. The ovnkube-trace utility runs ovn-trace, ovs-appctl ofproto/trace and ovn-detrace and correlates that information in a single output.

You can execute the ovnkube-trace binary from a dedicated container. For releases after OKD 4.7, you can also copy the binary to a local host and execute it from that host.

The binaries in the Quay images do not currently work for Dual IP stack or IPv6 only environments. For those environments, you must build from source.

Installing the ovnkube-trace on local host

The ovnkube-trace tool traces packet simulations for arbitrary UDP or TCP traffic between points in an OVN-Kubernetes driven OKD cluster. Copy the ovnkube-trace binary to your local host making it available to run against the cluster.

Prerequisites

  • You installed the OpenShift CLI (oc).

  • You are logged in to the cluster with a user with cluster-admin privileges.

Procedure

  1. Create a pod variable by using the following command:

    1. $  POD=$(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-master -o name | head -1 | awk -F '/' '{print $NF}')

  2. Run the following command on your local host to copy the binary from the ovnkube-master pods:

    1. $  oc cp -n openshift-ovn-kubernetes $POD:/usr/bin/ovnkube-trace ovnkube-trace

  3. Make ovnkube-trace executable by running the following command:

    1. $  chmod +x ovnkube-trace

  4. Display the options available with ovnkube-trace by running the following command:

    1. $  ./ovnkube-trace -help

    Expected output

    1. I0111 15:05:27.973305  204872 ovs.go:90] Maximum command line arguments set to: 191102
    2. Usage of ./ovnkube-trace:
    3.   -dst string
    4.         dest: destination pod name
    5.   -dst-ip string
    6.         destination IP address (meant for tests to external targets)
    7.   -dst-namespace string
    8.         k8s namespace of dest pod (default "default")
    9.   -dst-port string
    10.         dst-port: destination port (default "80")
    11.   -kubeconfig string
    12.         absolute path to the kubeconfig file
    13.   -loglevel string
    14.         loglevel: klog level (default "0")
    15.   -ovn-config-namespace string
    16.         namespace used by ovn-config itself
    17.   -service string
    18.         service: destination service name
    19.   -skip-detrace
    20.         skip ovn-detrace command
    21.   -src string
    22.         src: source pod name
    23.   -src-namespace string
    24.         k8s namespace of source pod (default "default")
    25.   -tcp
    26.         use tcp transport protocol
    27.   -udp
    28.         use udp transport protocol

    The command-line arguments supported are familiar Kubernetes constructs, such as namespaces, pods, services so you do not need to find the MAC address, the IP address of the destination nodes, or the ICMP type.

    The log levels are:

    • 0 (minimal output)

    • 2 (more verbose output showing results of trace commands)

    • 5 (debug output)

Running ovnkube-trace

Run ovn-trace to simulate packet forwarding within an OVN logical network.

Prerequisites

  • You installed the OpenShift CLI (oc).

  • You are logged in to the cluster with a user with cluster-admin privileges.

  • You have installed ovnkube-trace on local host

Example: Testing that DNS resolution works from a deployed pod

This example illustrates how to test the DNS resolution from a deployed pod to the core DNS pod that runs in the cluster.

Procedure

  1. Start a web service in the default namespace by entering the following command:

    1. $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80

  2. List the pods running in the openshift-dns namespace:

    1. oc get pods -n openshift-dns

    Example output

    1. NAME                  READY   STATUS    RESTARTS   AGE
    2. dns-default-467qw     2/2     Running   0          49m
    3. dns-default-6prvx     2/2     Running   0          53m
    4. dns-default-fkqr8     2/2     Running   0          53m
    5. dns-default-qv2rg     2/2     Running   0          49m
    6. dns-default-s29vr     2/2     Running   0          49m
    7. dns-default-vdsbn     2/2     Running   0          53m
    8. node-resolver-6thtt   1/1     Running   0          53m
    9. node-resolver-7ksdn   1/1     Running   0          49m
    10. node-resolver-8sthh   1/1     Running   0          53m
    11. node-resolver-c5ksw   1/1     Running   0          50m
    12. node-resolver-gbvdp   1/1     Running   0          53m
    13. node-resolver-sxhkd   1/1     Running   0          50m

  3. Run the following ovn-kube-trace command to verify DNS resolution is working:

    1. $ ./ovnkube-trace \
    2.   -src-namespace default \ (1)
    3.   -src web \ (2)
    4.   -dst-namespace openshift-dns \ (3)
    5.   -dst dns-default-467qw \ (4)
    6.   -udp -dst-port 53 \ (5)
    7.   -loglevel 0 (6)
    1Namespace of the source pod
    2Source pod name
    3Namespace of destination pod
    4Destination pod name
    5Use the udp transport protocol. Port 53 is the port the DNS service uses.
    6Set the log level to 1 (0 is minimal and 5 is debug)

    Expected output

    1. I0116 10:19:35.601303   17900 ovs.go:90] Maximum command line arguments set to: 191102
    2. ovn-trace source pod to destination pod indicates success from web to dns-default-467qw
    3. ovn-trace destination pod to source pod indicates success from dns-default-467qw to web
    4. ovs-appctl ofproto/trace source pod to destination pod indicates success from web to dns-default-467qw
    5. ovs-appctl ofproto/trace destination pod to source pod indicates success from dns-default-467qw to web
    6. ovn-detrace source pod to destination pod indicates success from web to dns-default-467qw
    7. ovn-detrace destination pod to source pod indicates success from dns-default-467qw to web

    The ouput indicates success from the deployed pod to the DNS port and also indicates that it is successful going back in the other direction. So you know bi-directional traffic is supported on UDP port 53 if my web pod wants to do dns resolution from core DNS.

If for example that did not work and you wanted to get the ovn-trace, the ovs-appctl ofproto/trace and ovn-detrace, and more debug type information increase the log level to 2 and run the command again as follows:

  1. $ ./ovnkube-trace \
  2.   -src-namespace default \
  3.   -src web \
  4.   -dst-namespace openshift-dns \
  5.   -dst dns-default-467qw \
  6.   -udp -dst-port 53 \
  7.   -loglevel 2

The output from this increased log level is too much to list here. In a failure situation the output of this command shows which flow is dropping that traffic. For example an egress or ingress network policy may be configured on the cluster that does not allow that traffic.

Example: Verifying by using debug output a configured default deny

This example illustrates how to identify by using the debug output that an ingress default deny policy blocks traffic.

Procedure

  1. Create the following YAML that defines a deny-by-default policy to deny ingress from all pods in all namespaces. Save the YAML in the deny-by-default.yaml file:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.   name: deny-by-default
    5.   namespace: default
    6. spec:
    7.   podSelector: {}
    8.   ingress: []

  2. Apply the policy by entering the following command:

    1. $ oc apply -f deny-by-default.yaml

    Example output

    1. networkpolicy.networking.k8s.io/deny-by-default created

  3. Start a web service in the default namespace by entering the following command:

    1. $ oc run web --namespace=default --image=nginx --labels="app=web" --expose --port=80

  4. Run the following command to create the prod namespace:

    1. $ oc create namespace prod

  5. Run the following command to label the prod namespace:

    1. $ oc label namespace/prod purpose=production

  6. Run the following command to deploy an alpine image in the prod namespace and start a shell:

    1. $ oc run test-6459 --namespace=prod --rm -i -t --image=alpine -- sh

  7. Open another terminal session.

  8. In this new terminal session run ovn-trace to verify the failure in communication between the source pod test-6459 running in namespace prod and destination pod running in the default namespace:

    1. $ ./ovnkube-trace \
    2.  -src-namespace prod \
    3.  -src test-6459 \
    4.  -dst-namespace default \
    5.  -dst web \
    6.  -tcp -dst-port 80 \
    7.  -loglevel 0

    Expected output

    1. I0116 14:20:47.380775   50822 ovs.go:90] Maximum command line arguments set to: 191102
    2. ovn-trace source pod to destination pod indicates failure from test-6459 to web

  9. Increase the log level to 2 to expose the reason for the failure by running the following command:

    1. $ ./ovnkube-trace \
    2.  -src-namespace prod \
    3.  -src test-6459 \
    4.  -dst-namespace default \
    5.  -dst web \
    6.  -tcp -dst-port 80 \
    7.  -loglevel 2

    Expected output

    1. ct_lb_mark /* default (use --ct to customize) */
    2. ------------------------------------------------
    3.  3. ls_out_acl_hint (northd.c:6092): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 32d45ad4
    4.     reg0[8] = 1;
    5.     reg0[10] = 1;
    6.     next;
    7.  4. ls_out_acl (northd.c:6435): reg0[10] == 1 && (outport == @a16982411286042166782_ingressDefaultDeny), priority 2000, uuid f730a887 (1)
    8.     ct_commit { ct_mark.blocked = 1; };
    1Ingress traffic is blocked due to the default deny policy being in place

  10. Create a policy that allows traffic from all pods in a particular namespaces with a label purpose=production. Save the YAML in the web-allow-prod.yaml file:

    1. kind: NetworkPolicy
    2. apiVersion: networking.k8s.io/v1
    3. metadata:
    4.  name: web-allow-prod
    5.  namespace: default
    6. spec:
    7.  podSelector:
    8.    matchLabels:
    9.      app: web
    10.  policyTypes:
    11.  - Ingress
    12.  ingress:
    13.  - from:
    14.    - namespaceSelector:
    15.        matchLabels:
    16.          purpose: production

  11. Apply the policy by entering the following command:

    1. $ oc apply -f web-allow-prod.yaml

  12. Run ovnkube-trace to verify that traffic is now allowed by entering the following command:

    1. $ ./ovnkube-trace \
    2. -src-namespace prod \
    3. -src test-6459 \
    4. -dst-namespace default \
    5. -dst web \
    6. -tcp -dst-port 80 \
    7. -loglevel 0

    Expected output

    1. I0116 14:25:44.055207   51695 ovs.go:90] Maximum command line arguments set to: 191102
    2. ovn-trace source pod to destination pod indicates success from test-6459 to web
    3. ovn-trace destination pod to source pod indicates success from web to test-6459
    4. ovs-appctl ofproto/trace source pod to destination pod indicates success from test-6459 to web
    5. ovs-appctl ofproto/trace destination pod to source pod indicates success from web to test-6459
    6. ovn-detrace source pod to destination pod indicates success from test-6459 to web
    7. ovn-detrace destination pod to source pod indicates success from web to test-6459

  13. In the open shell run the following command:

    1. wget -qO- --timeout=2 http://web.default

    Expected output

    1. <!DOCTYPE html>
    2. <html>
    3. <head>
    4. <title>Welcome to nginx!</title>
    5. <style>
    6. html { color-scheme: light dark; }
    7. body { width: 35em; margin: 0 auto;
    8. font-family: Tahoma, Verdana, Arial, sans-serif; }
    9. </style>
    10. </head>
    11. <body>
    12. <h1>Welcome to nginx!</h1>
    13. <p>If you see this page, the nginx web server is successfully installed and
    14. working. Further configuration is required.</p>
    16. <p>For online documentation and support please refer to
    17. <a href="http://nginx.org/">nginx.org</a>.<br/>
    18. Commercial support is available at
    19. <a href="http://nginx.com/">nginx.com</a>.</p>
    21. <p><em>Thank you for using nginx.</em></p>
    22. </body>
    23. </html>

Additional resources