Configuring an additional network

As a cluster administrator, you can configure an additional network for your cluster. The following network types are supported:

Approaches to managing an additional network

You can manage the life cycle of an additional network by two approaches. Each approach is mutually exclusive and you can only use one approach for managing an additional network at a time. For either approach, the additional network is managed by a Container Network Interface (CNI) plug-in that you configure.

For an additional network, IP addresses are provisioned through an IP Address Management (IPAM) CNI plug-in that you configure as part of the additional network. The IPAM plug-in supports a variety of IP address assignment approaches including DHCP and static assignment.

  • Modify the Cluster Network Operator (CNO) configuration: The CNO automatically creates and manages the NetworkAttachmentDefinition object. In addition to managing the object lifecycle the CNO ensures a DHCP is available for an additional network that uses a DHCP assigned IP address.

  • Applying a YAML manifest: You can manage the additional network directly by creating an NetworkAttachmentDefinition object. This approach allows for the chaining of CNI plug-ins.

Configuration for an additional network attachment

An additional network is configured via the NetworkAttachmentDefinition API in the k8s.cni.cncf.io API group.

Do not store any sensitive information or a secret in the NetworkAttachmentDefinition object because this information is accessible by the project administration user.

The configuration for the API is described in the following table:

Table 1. NetworkAttachmentDefinition API fields
FieldTypeDescription

metadata.name

string

The name for the additional network.

metadata.namespace

string

The namespace that the object is associated with.

spec.config

string

The CNI plug-in configuration in JSON format.

Configuration of an additional network through the Cluster Network Operator

The configuration for an additional network attachment is specified as part of the Cluster Network Operator (CNO) configuration.

The following YAML describes the configuration parameters for managing an additional network with the CNO:

Cluster Network Operator configuration

  1. apiVersion: operator.openshift.io/v1
  2. kind: Network
  3. metadata:
  4. name: cluster
  5. spec:
  6. # ...
  7. additionalNetworks: (1)
  8. - name: <name> (2)
  9. namespace: <namespace> (3)
  10. rawCNIConfig: |- (4)
  11. {
  12. ...
  13. }
  14. type: Raw
1An array of one or more additional network configurations.
2The name for the additional network attachment that you are creating. The name must be unique within the specified namespace.
3The namespace to create the network attachment in. If you do not specify a value, then the default namespace is used.
4A CNI plug-in configuration in JSON format.

Configuration of an additional network from a YAML manifest

The configuration for an additional network is specified from a YAML configuration file, such as in the following example:

  1. apiVersion: k8s.cni.cncf.io/v1
  2. kind: NetworkAttachmentDefinition
  3. metadata:
  4. name: <name> (1)
  5. spec:
  6. config: |- (2)
  7. {
  8. ...
  9. }
1The name for the additional network attachment that you are creating.
2A CNI plug-in configuration in JSON format.

Configurations for additional network types

The specific configuration fields for additional networks is described in the following sections.

Configuration for a bridge additional network

The following object describes the configuration parameters for the bridge CNI plug-in:

Table 2. Bridge CNI plug-in JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

bridge

string

Specify the name of the virtual bridge to use. If the bridge interface does not exist on the host, it is created. The default value is cni0.

ipam

object

The configuration object for the IPAM CNI plug-in. The plug-in manages IP address assignment for the attachment definition.

ipMasq

boolean

Set to true to enable IP masquerading for traffic that leaves the virtual network. The source IP address for all traffic is rewritten to the bridge’s IP address. If the bridge does not have an IP address, this setting has no effect. The default value is false.

isGateway

boolean

Set to true to assign an IP address to the bridge. The default value is false.

isDefaultGateway

boolean

Set to true to configure the bridge as the default gateway for the virtual network. The default value is false. If isDefaultGateway is set to true, then isGateway is also set to true automatically.

forceAddress

boolean

Set to true to allow assignment of a previously assigned IP address to the virtual bridge. When set to false, if an IPv4 address or an IPv6 address from overlapping subsets is assigned to the virtual bridge, an error occurs. The default value is false.

hairpinMode

boolean

Set to true to allow the virtual bridge to send an Ethernet frame back through the virtual port it was received on. This mode is also known as reflective relay. The default value is false.

promiscMode

boolean

Set to true to enable promiscuous mode on the bridge. The default value is false.

vlan

string

Specify a virtual LAN (VLAN) tag as an integer value. By default, no VLAN tag is assigned.

mtu

string

Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

bridge configuration example

The following example configures an additional network named bridge-net:

  1. {
  2. "cniVersion": "0.3.1",
  3. "name": "work-network",
  4. "type": "bridge",
  5. "isGateway": true,
  6. "vlan": 2,
  7. "ipam": {
  8. "type": "dhcp"
  9. }
  10. }

Configuration for a host device additional network

Specify your network device by setting only one of the following parameters: device, hwaddr, kernelpath, or pciBusID.

The following object describes the configuration parameters for the host-device CNI plug-in:

Table 3. Host device CNI plug-in JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plug-in to configure: host-device.

device

string

Optional: The name of the device, such as eth0.

hwaddr

string

Optional: The device hardware MAC address.

kernelpath

string

Optional: The Linux kernel device path, such as /sys/devices/pci0000:00/0000:00:1f.6.

pciBusID

string

Optional: The PCI address of the network device, such as 0000:00:1f.6.

host-device configuration example

The following example configures an additional network named hostdev-net:

  1. {
  2. "cniVersion": "0.3.1",
  3. "name": "work-network",
  4. "type": "host-device",
  5. "device": "eth1"
  6. }

Configuration for an IPVLAN additional network

The following object describes the configuration parameters for the IPVLAN CNI plug-in:

Table 4. IPVLAN CNI plug-in JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plug-in to configure: ipvlan.

mode

string

The operating mode for the virtual network. The value must be l2, l3, or l3s. The default value is l2.

master

string

The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

mtu

integer

Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

ipam

object

The configuration object for the IPAM CNI plug-in. The plug-in manages IP address assignment for the attachment definition.

Do not specify dhcp. Configuring IPVLAN with DHCP is not supported because IPVLAN interfaces share the MAC address with the host interface.

ipvlan configuration example

The following example configures an additional network named ipvlan-net:

  1. {
  2. "cniVersion": "0.3.1",
  3. "name": "work-network",
  4. "type": "ipvlan",
  5. "master": "eth1",
  6. "mode": "l3",
  7. "ipam": {
  8. "type": "static",
  9. "addresses": [
  10. {
  11. "address": "192.168.10.10"
  12. }
  13. ]
  14. }
  15. }

Configuration for a MACVLAN additional network

The following object describes the configuration parameters for the macvlan CNI plug-in:

Table 5. MACVLAN CNI plug-in JSON configuration object
FieldTypeDescription

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plug-in to configure: macvlan.

mode

string

Configures traffic visibility on the virtual network. Must be either bridge, passthru, private, or vepa. If a value is not provided, the default value is bridge.

master

string

The Ethernet, bonded, or VLAN interface to associate with the virtual interface. If a value is not specified, then the host system’s primary Ethernet interface is used.

mtu

string

The maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

ipam

object

The configuration object for the IPAM CNI plug-in. The plug-in manages IP address assignment for the attachment definition.

macvlan configuration example

The following example configures an additional network named macvlan-net:

  1. {
  2. "cniVersion": "0.3.1",
  3. "name": "macvlan-net",
  4. "type": "macvlan",
  5. "master": "eth1",
  6. "mode": "bridge",
  7. "ipam": {
  8. "type": "dhcp"
  9. }
  10. }

Configuration of IP address assignment for an additional network

The IP address management (IPAM) Container Network Interface (CNI) plug-in provides IP addresses for other CNI plug-ins.

You can use the following IP address assignment types:

  • Static assignment.

  • Dynamic assignment through a DHCP server. The DHCP server you specify must be reachable from the additional network.

  • Dynamic assignment through the Whereabouts IPAM CNI plug-in.

Static IP address assignment configuration

The following table describes the configuration for static IP address assignment:

Table 6. ipam static configuration object
FieldTypeDescription

type

string

The IPAM address type. The value static is required.

addresses

array

An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported.

routes

array

An array of objects specifying routes to configure inside the pod.

dns

array

Optional: An array of objects specifying the DNS configuration.

The addresses array requires objects with the following fields:

Table 7. ipam.addresses[] array
FieldTypeDescription

address

string

An IP address and network prefix that you specify. For example, if you specify 10.10.21.10/24, then the additional network is assigned an IP address of 10.10.21.10 and the netmask is 255.255.255.0.

gateway

string

The default gateway to route egress network traffic to.

Table 8. ipam.routes[] array
FieldTypeDescription

dst

string

The IP address range in CIDR format, such as 192.168.17.0/24 or 0.0.0.0/0 for the default route.

gw

string

The gateway where network traffic is routed.

Table 9. ipam.dns object
FieldTypeDescription

nameservers

array

An array of one or more IP addresses for to send DNS queries to.

domain

array

The default domain to append to a hostname. For example, if the domain is set to example.com, a DNS lookup query for example-host is rewritten as example-host.example.com.

search

array

An array of domain names to append to an unqualified hostname, such as example-host, during a DNS lookup query.

Static IP address assignment configuration example

  1. {
  2. "ipam": {
  3. "type": "static",
  4. "addresses": [
  5. {
  6. "address": "191.168.1.7"
  7. }
  8. ]
  9. }
  10. }

Dynamic IP address (DHCP) assignment configuration

The following JSON describes the configuration for dynamic IP address address assignment with DHCP.

Renewal of DHCP leases

A pod obtains its original DHCP lease when it is created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.

To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:

Example shim network attachment definition
  1. apiVersion: operator.openshift.io/v1
  2. kind: Network
  3. metadata:
  4. name: cluster
  5. spec:
  6. additionalNetworks:
  7. - name: dhcp-shim
  8. namespace: default
  9. type: Raw
  10. rawCNIConfig: |-
  11. {
  12. name”: dhcp-shim”,
  13. cniVersion”: 0.3.1”,
  14. type”: bridge”,
  15. ipam”: {
  16. type”: dhcp
  17. }
  18. }
  19. # …
Table 10. ipam DHCP configuration object
FieldTypeDescription

type

string

The IPAM address type. The value dhcp is required.

Dynamic IP address (DHCP) assignment configuration example

  1. {
  2. "ipam": {
  3. "type": "dhcp"
  4. }
  5. }

Dynamic IP address assignment configuration with Whereabouts

The Whereabouts CNI plug-in allows the dynamic assignment of an IP address to an additional network without the use of a DHCP server.

The following table describes the configuration for dynamic IP address assignment with Whereabouts:

Table 11. ipam whereabouts configuration object
FieldTypeDescription

type

string

The IPAM address type. The value whereabouts is required.

range

string

An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses.

exclude

array

Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned.

Dynamic IP address assignment configuration example that uses Whereabouts

  1. {
  2. "ipam": {
  3. "type": "whereabouts",
  4. "range": "192.0.2.192/27",
  5. "exclude": [
  6. "192.0.2.192/30",
  7. "192.0.2.196/32"
  8. ]
  9. }
  10. }

Creating an additional network attachment with the Cluster Network Operator

The Cluster Network Operator (CNO) manages additional network definitions. When you specify an additional network to create, the CNO creates the NetworkAttachmentDefinition object automatically.

Do not edit the NetworkAttachmentDefinition objects that the Cluster Network Operator manages. Doing so might disrupt network traffic on your additional network.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Optional: Create the namespace for the additional networks:

    1. $ oc create namespace <namespace_name>
  2. To edit the CNO configuration, enter the following command:

    1. $ oc edit networks.operator.openshift.io cluster
  3. Modify the CR that you are creating by adding the configuration for the additional network that you are creating, as in the following example CR.

    1. apiVersion: operator.openshift.io/v1
    2. kind: Network
    3. metadata:
    4. name: cluster
    5. spec:
    6. # ...
    7. additionalNetworks:
    8. - name: tertiary-net
    9. namespace: namespace2
    10. type: Raw
    11. rawCNIConfig: |-
    12. {
    13. "cniVersion": "0.3.1",
    14. "name": "tertiary-net",
    15. "type": "ipvlan",
    16. "master": "eth1",
    17. "mode": "l2",
    18. "ipam": {
    19. "type": "static",
    20. "addresses": [
    21. {
    22. "address": "192.168.1.23/24"
    23. }
    24. ]
    25. }
    26. }
  4. Save your changes and quit the text editor to commit your changes.

Verification

  • Confirm that the CNO created the NetworkAttachmentDefinition object by running the following command. There might be a delay before the CNO creates the object.

    1. $ oc get network-attachment-definitions -n <namespace>

    where:

    <namespace>

    Specifies the namespace for the network attachment that you added to the CNO configuration.

    Example output

    1. NAME AGE
    2. test-network-1 14m

Creating an additional network attachment by applying a YAML manifest

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a YAML file with your additional network configuration, such as in the following example:

    1. apiVersion: k8s.cni.cncf.io/v1
    2. kind: NetworkAttachmentDefinition
    3. metadata:
    4. name: next-net
    5. spec:
    6. config: |-
    7. {
    8. "cniVersion": "0.3.1",
    9. "name": "work-network",
    10. "type": "host-device",
    11. "device": "eth1",
    12. "ipam": {
    13. "type": "dhcp"
    14. }
    15. }
  2. To create the additional network, enter the following command:

    1. $ oc apply -f <file>.yaml

    where:

    <file>

    Specifies the name of the file contained the YAML manifest.