Logging for egress firewall and network policy rules

As a cluster administrator, you can configure audit logging for your cluster and enable logging for one or more namespaces. OKD produces audit logs for both egress firewalls and network policies.

Audit logging is available for only the OVN-Kubernetes network plugin.

Audit logging

The OVN-Kubernetes network plugin uses Open Virtual Network (OVN) ACLs to manage egress firewalls and network policies. Audit logging exposes allow and deny ACL events.

You can configure the destination for audit logs, such as a syslog server or a UNIX domain socket. Regardless of any additional configuration, an audit log is always saved to /var/log/ovn/acl-audit-log.log on each OVN-Kubernetes pod in the cluster.

Audit logging is enabled per namespace by annotating the namespace with the k8s.ovn.org/acl-logging key as in the following example:

Example namespace annotation

  1. kind: Namespace
  2. apiVersion: v1
  3. metadata:
  4. name: example1
  5. annotations:
  6. k8s.ovn.org/acl-logging: |-
  7. {
  8. "deny": "info",
  9. "allow": "info"
  10. }

The logging format is compatible with syslog as defined by RFC5424. The syslog facility is configurable and defaults to local0. An example log entry might resemble the following:

Example ACL deny log entry for a network policy

  1. 2023-11-02T16:28:54.139Z|00004|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
  2. 2023-11-02T16:28:55.187Z|00005|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
  3. 2023-11-02T16:28:57.235Z|00006|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn

The following table describes namespace annotation values:

Table 1. Audit logging namespace annotation
AnnotationValue

k8s.ovn.org/acl-logging

You must specify at least one of allow, deny, or both to enable audit logging for a namespace.

    deny

    Optional: Specify alert, warning, notice, info, or debug.

    allow

    Optional: Specify alert, warning, notice, info, or debug.

Audit configuration

The configuration for audit logging is specified as part of the OVN-Kubernetes cluster network provider configuration. The following YAML illustrates the default values for the audit logging:

Audit logging configuration

  1. apiVersion: operator.openshift.io/v1
  2. kind: Network
  3. metadata:
  4. name: cluster
  5. spec:
  6. defaultNetwork:
  7. ovnKubernetesConfig:
  8. policyAuditConfig:
  9. destination: "null"
  10. maxFileSize: 50
  11. rateLimit: 20
  12. syslogFacility: local0

The following table describes the configuration fields for audit logging.

Table 2. policyAuditConfig object
FieldTypeDescription

rateLimit

integer

The maximum number of messages to generate every second per node. The default value is 20 messages per second.

maxFileSize

integer

The maximum size for the audit log in bytes. The default value is 50000000 or 50 MB.

destination

string

One of the following additional audit log targets:

    libc

    The libc syslog() function of the journald process on the host.

    udp:<host>:<port>

    A syslog server. Replace <host>:<port> with the host and port of the syslog server.

    unix:<file>

    A Unix Domain Socket file specified by <file>.

    null

    Do not send the audit logs to any additional target.

syslogFacility

string

The syslog facility, such as kern, as defined by RFC5424. The default value is local0.

Configuring egress firewall and network policy auditing for a cluster

As a cluster administrator, you can customize audit logging for your cluster.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in to the cluster with a user with cluster-admin privileges.

Procedure

  • To customize the audit logging configuration, enter the following command:

    1. $ oc edit network.operator.openshift.io/cluster

    You can alternatively customize and apply the following YAML to configure audit logging:

    1. apiVersion: operator.openshift.io/v1
    2. kind: Network
    3. metadata:
    4. name: cluster
    5. spec:
    6. defaultNetwork:
    7. ovnKubernetesConfig:
    8. policyAuditConfig:
    9. destination: null
    10. maxFileSize: 50
    11. rateLimit: 20
    12. syslogFacility: local0

Verification

  1. To create a namespace with network policies complete the following steps:

    1. Create a namespace for verification:

      1. $ cat <<EOF| oc create -f -
      2. kind: Namespace
      3. apiVersion: v1
      4. metadata:
      5. name: verify-audit-logging
      6. annotations:
      7. k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'
      8. EOF

      Example output

      1. namespace/verify-audit-logging created
    2. Create network policies for the namespace:

      1. $ cat <<EOF| oc create -n verify-audit-logging -f -
      2. apiVersion: networking.k8s.io/v1
      3. kind: NetworkPolicy
      4. metadata:
      5. name: deny-all
      6. spec:
      7. podSelector:
      8. matchLabels:
      9. policyTypes:
      10. - Ingress
      11. - Egress
      12. ---
      13. apiVersion: networking.k8s.io/v1
      14. kind: NetworkPolicy
      15. metadata:
      16. name: allow-from-same-namespace
      17. namespace: verify-audit-logging
      18. spec:
      19. podSelector: {}
      20. policyTypes:
      21. - Ingress
      22. - Egress
      23. ingress:
      24. - from:
      25. - podSelector: {}
      26. egress:
      27. - to:
      28. - namespaceSelector:
      29. matchLabels:
      30. kubernetes.io/metadata.name: verify-audit-logging
      31. EOF

      Example output

      1. networkpolicy.networking.k8s.io/deny-all created
      2. networkpolicy.networking.k8s.io/allow-from-same-namespace created
  2. Create a pod for source traffic in the default namespace:

    1. $ cat <<EOF| oc create -n default -f -
    2. apiVersion: v1
    3. kind: Pod
    4. metadata:
    5. name: client
    6. spec:
    7. containers:
    8. - name: client
    9. image: registry.access.redhat.com/rhel7/rhel-tools
    10. command: ["/bin/sh", "-c"]
    11. args:
    12. ["sleep inf"]
    13. EOF
  3. Create two pods in the verify-audit-logging namespace:

    1. $ for name in client server; do
    2. cat <<EOF| oc create -n verify-audit-logging -f -
    3. apiVersion: v1
    4. kind: Pod
    5. metadata:
    6. name: ${name}
    7. spec:
    8. containers:
    9. - name: ${name}
    10. image: registry.access.redhat.com/rhel7/rhel-tools
    11. command: ["/bin/sh", "-c"]
    12. args:
    13. ["sleep inf"]
    14. EOF
    15. done

    Example output

    1. pod/client created
    2. pod/server created
  4. To generate traffic and produce network policy audit log entries, complete the following steps:

    1. Obtain the IP address for pod named server in the verify-audit-logging namespace:

      1. $ POD_IP=$(oc get pods server -n verify-audit-logging -o jsonpath='{.status.podIP}')
    2. Ping the IP address from the previous command from the pod named client in the default namespace and confirm that all packets are dropped:

      1. $ oc exec -it client -n default -- /bin/ping -c 2 $POD_IP

      Example output

      1. PING 10.128.2.55 (10.128.2.55) 56(84) bytes of data.
      2. --- 10.128.2.55 ping statistics ---
      3. 2 packets transmitted, 0 received, 100% packet loss, time 2041ms
    3. Ping the IP address saved in the POD_IP shell environment variable from the pod named client in the verify-audit-logging namespace and confirm that all packets are allowed:

      1. $ oc exec -it client -n verify-audit-logging -- /bin/ping -c 2 $POD_IP

      Example output

      1. PING 10.128.0.86 (10.128.0.86) 56(84) bytes of data.
      2. 64 bytes from 10.128.0.86: icmp_seq=1 ttl=64 time=2.21 ms
      3. 64 bytes from 10.128.0.86: icmp_seq=2 ttl=64 time=0.440 ms
      4. --- 10.128.0.86 ping statistics ---
      5. 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      6. rtt min/avg/max/mdev = 0.440/1.329/2.219/0.890 ms
  5. Display the latest entries in the network policy audit log:

    1. $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
    2. oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
    3. done

    Example output

    1. 2023-11-02T16:28:54.139Z|00004|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
    2. 2023-11-02T16:28:55.187Z|00005|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
    3. 2023-11-02T16:28:57.235Z|00006|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:Ingress", verdict=drop, severity=alert, direction=to-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:01,dl_dst=0a:58:0a:81:02:23,nw_src=10.131.0.39,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=62,nw_frag=no,tp_src=58496,tp_dst=8080,tcp_flags=syn
    4. 2023-11-02T16:49:57.909Z|00028|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    5. 2023-11-02T16:49:57.909Z|00029|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    6. 2023-11-02T16:49:58.932Z|00030|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    7. 2023-11-02T16:49:58.932Z|00031|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0

Enabling egress firewall and network policy audit logging for a namespace

As a cluster administrator, you can enable audit logging for a namespace.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in to the cluster with a user with cluster-admin privileges.

Procedure

  • To enable audit logging for a namespace, enter the following command:

    1. $ oc annotate namespace <namespace> \
    2. k8s.ovn.org/acl-logging='{ "deny": "alert", "allow": "notice" }'

    where:

    <namespace>

    Specifies the name of the namespace.

    You can alternatively apply the following YAML to enable audit logging:

    1. kind: Namespace
    2. apiVersion: v1
    3. metadata:
    4. name: <namespace>
    5. annotations:
    6. k8s.ovn.org/acl-logging: |-
    7. {
    8. deny”: alert”,
    9. allow”: notice
    10. }

    Example output

    1. namespace/verify-audit-logging annotated

Verification

  • Display the latest entries in the audit log:

    1. $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
    2. oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
    3. done

    Example output

    1. 2023-11-02T16:49:57.909Z|00028|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    2. 2023-11-02T16:49:57.909Z|00029|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    3. 2023-11-02T16:49:58.932Z|00030|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Egress:0", verdict=allow, severity=alert, direction=from-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0
    4. 2023-11-02T16:49:58.932Z|00031|acl_log(ovn_pinctrl0)|INFO|name="NP:verify-audit-logging:allow-from-same-namespace:Ingress:0", verdict=allow, severity=alert, direction=to-lport: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:81:02:22,dl_dst=0a:58:0a:81:02:23,nw_src=10.129.2.34,nw_dst=10.129.2.35,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,icmp_type=8,icmp_code=0

Disabling egress firewall and network policy audit logging for a namespace

As a cluster administrator, you can disable audit logging for a namespace.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in to the cluster with a user with cluster-admin privileges.

Procedure

  • To disable audit logging for a namespace, enter the following command:

    1. $ oc annotate --overwrite namespace <namespace> k8s.ovn.org/acl-logging-

    where:

    <namespace>

    Specifies the name of the namespace.

    You can alternatively apply the following YAML to disable audit logging:

    1. kind: Namespace
    2. apiVersion: v1
    3. metadata:
    4. name: <namespace>
    5. annotations:
    6. k8s.ovn.org/acl-logging: null

    Example output

    1. namespace/verify-audit-logging annotated

Additional resources