Cluster Network Operator in OKD

The Cluster Network Operator (CNO) deploys and manages the cluster network components on an OKD cluster, including the Container Network Interface (CNI) network plugin selected for the cluster during installation.

Cluster Network Operator

The Cluster Network Operator implements the network API from the operator.openshift.io API group. The Operator deploys the OVN-Kubernetes network plugin, or the network provider plugin that you selected during cluster installation, by using a daemon set.

Procedure

The Cluster Network Operator is deployed during installation as a Kubernetes Deployment.

  1. Run the following command to view the Deployment status:

    1. $ oc get -n openshift-network-operator deployment/network-operator

    Example output

    1. NAME READY UP-TO-DATE AVAILABLE AGE
    2. network-operator 1/1 1 1 56m
  2. Run the following command to view the state of the Cluster Network Operator:

    1. $ oc get clusteroperator/network

    Example output

    1. NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
    2. network 4.5.4 True False False 50m

    The following fields provide information about the status of the operator: AVAILABLE, PROGRESSING, and DEGRADED. The AVAILABLE field is True when the Cluster Network Operator reports an available status condition.

Viewing the cluster network configuration

Every new OKD installation has a network.config object named cluster.

Procedure

  • Use the oc describe command to view the cluster network configuration:

    1. $ oc describe network.config/cluster

    Example output

    1. Name: cluster
    2. Namespace:
    3. Labels: <none>
    4. Annotations: <none>
    5. API Version: config.openshift.io/v1
    6. Kind: Network
    7. Metadata:
    8. Self Link: /apis/config.openshift.io/v1/networks/cluster
    9. Spec: (1)
    10. Cluster Network:
    11. Cidr: 10.128.0.0/14
    12. Host Prefix: 23
    13. Network Type: OpenShiftSDN
    14. Service Network:
    15. 172.30.0.0/16
    16. Status: (2)
    17. Cluster Network:
    18. Cidr: 10.128.0.0/14
    19. Host Prefix: 23
    20. Cluster Network MTU: 8951
    21. Network Type: OpenShiftSDN
    22. Service Network:
    23. 172.30.0.0/16
    24. Events: <none>
    1The Spec field displays the configured state of the cluster network.
    2The Status field displays the current state of the cluster network configuration.

Viewing Cluster Network Operator status

You can inspect the status and view the details of the Cluster Network Operator using the oc describe command.

Procedure

  • Run the following command to view the status of the Cluster Network Operator:

    1. $ oc describe clusteroperators/network

Viewing Cluster Network Operator logs

You can view Cluster Network Operator logs by using the oc logs command.

Procedure

  • Run the following command to view the logs of the Cluster Network Operator:

    1. $ oc logs --namespace=openshift-network-operator deployment/network-operator

Cluster Network Operator configuration

The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO) configuration and stored in a custom resource (CR) object that is named cluster. The CR specifies the fields for the Network API in the operator.openshift.io API group.

The CNO configuration inherits the following fields during cluster installation from the Network API in the Network.config.openshift.io API group and these fields cannot be changed:

clusterNetwork

IP address pools from which pod IP addresses are allocated.

serviceNetwork

IP address pool for services.

defaultNetwork.type

Cluster network plugin, such as OpenShift SDN or OVN-Kubernetes.

After cluster installation, you cannot modify the fields listed in the previous section.

You can specify the cluster network plugin configuration for your cluster by setting the fields for the defaultNetwork object in the CNO object named cluster.

Cluster Network Operator configuration object

The fields for the Cluster Network Operator (CNO) are described in the following table:

Table 1. Cluster Network Operator configuration object
FieldTypeDescription

metadata.name

string

The name of the CNO object. This name is always cluster.

spec.clusterNetwork

array

A list specifying the blocks of IP addresses from which pod IP addresses are allocated and the subnet prefix length assigned to each individual node in the cluster. For example:

  1. spec:
  2. clusterNetwork:
  3. - cidr: 10.128.0.0/19
  4. hostPrefix: 23
  5. - cidr: 10.128.32.0/19
  6. hostPrefix: 23

This value is ready-only and inherited from the Network.config.openshift.io object named cluster during cluster installation.

spec.serviceNetwork

array

A block of IP addresses for services. The OpenShift SDN and OVN-Kubernetes network plugins support only a single IP address block for the service network. For example:

  1. spec:
  2. serviceNetwork:
  3. - 172.30.0.0/14

This value is ready-only and inherited from the Network.config.openshift.io object named cluster during cluster installation.

spec.defaultNetwork

object

Configures the network plugin for the cluster network.

spec.kubeProxyConfig

object

The fields for this object specify the kube-proxy configuration. If you are using the OVN-Kubernetes cluster network plugin, the kube-proxy configuration has no effect.

defaultNetwork object configuration

The values for the defaultNetwork object are defined in the following table:

Table 2. defaultNetwork object
FieldTypeDescription

type

string

Either OpenShiftSDN or OVNKubernetes. The Red Hat OpenShift Networking network plugin is selected during installation. This value cannot be changed after cluster installation.

OKD uses the OVN-Kubernetes network plugin by default.

openshiftSDNConfig

object

This object is only valid for the OpenShift SDN network plugin.

ovnKubernetesConfig

object

This object is only valid for the OVN-Kubernetes network plugin.

Configuration for the OpenShift SDN network plugin

The following table describes the configuration fields for the OpenShift SDN network plugin:

Table 3. openshiftSDNConfig object
FieldTypeDescription

mode

string

The network isolation mode for OpenShift SDN.

mtu

integer

The maximum transmission unit (MTU) for the VXLAN overlay network. This value is normally configured automatically.

vxlanPort

integer

The port to use for all VXLAN packets. The default value is 4789.

You can only change the configuration for your cluster network plugin during cluster installation.

Example OpenShift SDN configuration

  1. defaultNetwork:
  2. type: OpenShiftSDN
  3. openshiftSDNConfig:
  4. mode: NetworkPolicy
  5. mtu: 1450
  6. vxlanPort: 4789
Configuration for the OVN-Kubernetes network plugin

The following table describes the configuration fields for the OVN-Kubernetes network plugin:

Table 4. ovnKubernetesConfig object
FieldTypeDescription

mtu

integer

The maximum transmission unit (MTU) for the Geneve (Generic Network Virtualization Encapsulation) overlay network. This value is normally configured automatically.

genevePort

integer

The UDP port for the Geneve overlay network.

ipsecConfig

object

If the field is present, IPsec is enabled for the cluster.

policyAuditConfig

object

Specify a configuration object for customizing network policy audit logging. If unset, the defaults audit log settings are used.

gatewayConfig

object

Optional: Specify a configuration object for customizing how egress traffic is sent to the node gateway.

  1. While migrating egress traffic, you can expect some disruption to workloads and service traffic until the Cluster Network Operator (CNO) successfully rolls out the changes.

v4InternalSubnet

If your existing network infrastructure overlaps with the 100.64.0.0/16 IPv4 subnet, you can specify a different IP address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster.

For example, if the clusterNetwork.cidr is 10.128.0.0/14 and the clusterNetwork.hostPrefix is /23, then the maximum number of nodes is 2^(23-14)=128. An IP address is also required for the gateway, network, and broadcast addresses. Therefore the internal IP address range must be at least a /24.

This field cannot be changed after installation.

The default value is 100.64.0.0/16.

v6InternalSubnet

If your existing network infrastructure overlaps with the fd98::/48 IPv6 subnet, you can specify a different IP address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster.

This field cannot be changed after installation.

The default value is fd98::/48.

Table 5. policyAuditConfig object
FieldTypeDescription

rateLimit

integer

The maximum number of messages to generate every second per node. The default value is 20 messages per second.

maxFileSize

integer

The maximum size for the audit log in bytes. The default value is 50000000 or 50 MB.

destination

string

One of the following additional audit log targets:

    libc

    The libc syslog() function of the journald process on the host.

    udp:<host>:<port>

    A syslog server. Replace <host>:<port> with the host and port of the syslog server.

    unix:<file>

    A Unix Domain Socket file specified by <file>.

    null

    Do not send the audit logs to any additional target.

syslogFacility

string

The syslog facility, such as kern, as defined by RFC5424. The default value is local0.

Table 6. gatewayConfig object
FieldTypeDescription

routingViaHost

boolean

Set this field to true to send egress traffic from pods to the host networking stack. For highly-specialized installations and applications that rely on manually configured routes in the kernel routing table, you might want to route egress traffic to the host networking stack. By default, egress traffic is processed in OVN to exit the cluster and is not affected by specialized routes in the kernel routing table. The default value is false.

This field has an interaction with the Open vSwitch hardware offloading feature. If you set this field to true, you do not receive the performance benefits of the offloading because egress traffic is processed by the host networking stack.

You can only change the configuration for your cluster network plugin during cluster installation, except for the gatewayConfig field that can be changed at runtime as a post-installation activity.

Example OVN-Kubernetes configuration with IPSec enabled

  1. defaultNetwork:
  2. type: OVNKubernetes
  3. ovnKubernetesConfig:
  4. mtu: 1400
  5. genevePort: 6081
  6. ipsecConfig: {}

kubeProxyConfig object configuration

The values for the kubeProxyConfig object are defined in the following table:

Table 7. kubeProxyConfig object
FieldTypeDescription

iptablesSyncPeriod

string

The refresh period for iptables rules. The default value is 30s. Valid suffixes include s, m, and h and are described in the Go time package documentation.

Because of performance improvements introduced in OKD 4.3 and greater, adjusting the iptablesSyncPeriod parameter is no longer necessary.

proxyArguments.iptables-min-sync-period

array

The minimum duration before refreshing iptables rules. This field ensures that the refresh does not happen too frequently. Valid suffixes include s, m, and h and are described in the Go time package. The default value is:

  1. kubeProxyConfig:
  2. proxyArguments:
  3. iptables-min-sync-period:
  4. - 0s

Cluster Network Operator example configuration

A complete CNO configuration is specified in the following example:

Example Cluster Network Operator object

  1. apiVersion: operator.openshift.io/v1
  2. kind: Network
  3. metadata:
  4. name: cluster
  5. spec:
  6. clusterNetwork: (1)
  7. - cidr: 10.128.0.0/14
  8. hostPrefix: 23
  9. serviceNetwork: (1)
  10. - 172.30.0.0/16
  11. defaultNetwork: (1)
  12. type: OpenShiftSDN
  13. openshiftSDNConfig:
  14. mode: NetworkPolicy
  15. mtu: 1450
  16. vxlanPort: 4789
  17. kubeProxyConfig:
  18. iptablesSyncPeriod: 30s
  19. proxyArguments:
  20. iptables-min-sync-period:
  21. - 0s
1Configured only during cluster installation.

Additional resources