Configuring an SR-IOV InfiniBand network attachment

You can configure an InfiniBand (IB) network attachment for an Single Root I/O Virtualization (SR-IOV) device in the cluster.

InfiniBand device configuration object

You can configure an InfiniBand (IB) network device by defining an SriovIBNetwork object.

The following YAML describes an SriovIBNetwork object:

  1. apiVersion: sriovnetwork.openshift.io/v1
  2. kind: SriovIBNetwork
  3. metadata:
  4. name: <name> (1)
  5. namespace: openshift-sriov-network-operator (2)
  6. spec:
  7. resourceName: <sriov_resource_name> (3)
  8. networkNamespace: <target_namespace> (4)
  9. ipam: |- (5)
  10. {}
  11. linkState: <link_state> (6)
  12. capabilities: <capabilities> (7)
1A name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with same name.
2The namespace where the SR-IOV Operator is installed.
3The value for the spec.resourceName parameter from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network.
4The target namespace for the SriovIBNetwork object. Only pods in the target namespace can attach to the network device.
5Optional: A configuration object for the IPAM CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.
6Optional: The link state of virtual function (VF). Allowed values are enable, disable and auto.
7Optional: The capabilities to configure for this network. You can specify “{ “ips”: true }” to enable IP address support or “{ “infinibandGUID”: true }” to enable IB Global Unique Identifier (GUID) support.

Configuration of IP address assignment for an additional network

The IP address management (IPAM) Container Network Interface (CNI) plugin provides IP addresses for other CNI plugins.

You can use the following IP address assignment types:

  • Static assignment.

  • Dynamic assignment through a DHCP server. The DHCP server you specify must be reachable from the additional network.

  • Dynamic assignment through the Whereabouts IPAM CNI plugin.

Static IP address assignment configuration

The following table describes the configuration for static IP address assignment:

Table 1. ipam static configuration object
FieldTypeDescription

type

string

The IPAM address type. The value static is required.

addresses

array

An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported.

routes

array

An array of objects specifying routes to configure inside the pod.

dns

array

Optional: An array of objects specifying the DNS configuration.

The addresses array requires objects with the following fields:

Table 2. ipam.addresses[] array
FieldTypeDescription

address

string

An IP address and network prefix that you specify. For example, if you specify 10.10.21.10/24, then the additional network is assigned an IP address of 10.10.21.10 and the netmask is 255.255.255.0.

gateway

string

The default gateway to route egress network traffic to.

Table 3. ipam.routes[] array
FieldTypeDescription

dst

string

The IP address range in CIDR format, such as 192.168.17.0/24 or 0.0.0.0/0 for the default route.

gw

string

The gateway where network traffic is routed.

Table 4. ipam.dns object
FieldTypeDescription

nameservers

array

An array of one or more IP addresses for to send DNS queries to.

domain

array

The default domain to append to a hostname. For example, if the domain is set to example.com, a DNS lookup query for example-host is rewritten as example-host.example.com.

search

array

An array of domain names to append to an unqualified hostname, such as example-host, during a DNS lookup query.

Static IP address assignment configuration example

  1. {
  2. "ipam": {
  3. "type": "static",
  4. "addresses": [
  5. {
  6. "address": "191.168.1.7/24"
  7. }
  8. ]
  9. }
  10. }

Dynamic IP address (DHCP) assignment configuration

The following JSON describes the configuration for dynamic IP address address assignment with DHCP.

Renewal of DHCP leases

A pod obtains its original DHCP lease when it is created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.

To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:

Example shim network attachment definition
  1. apiVersion: operator.openshift.io/v1
  2. kind: Network
  3. metadata:
  4. name: cluster
  5. spec:
  6. additionalNetworks:
  7. - name: dhcp-shim
  8. namespace: default
  9. type: Raw
  10. rawCNIConfig: |-
  11. {
  12. name”: dhcp-shim”,
  13. cniVersion”: 0.3.1”,
  14. type”: bridge”,
  15. ipam”: {
  16. type”: dhcp
  17. }
  18. }
  19. # …
Table 5. ipam DHCP configuration object
FieldTypeDescription

type

string

The IPAM address type. The value dhcp is required.

Dynamic IP address (DHCP) assignment configuration example

  1. {
  2. "ipam": {
  3. "type": "dhcp"
  4. }
  5. }

Dynamic IP address assignment configuration with Whereabouts

The Whereabouts CNI plugin allows the dynamic assignment of an IP address to an additional network without the use of a DHCP server.

The following table describes the configuration for dynamic IP address assignment with Whereabouts:

Table 6. ipam whereabouts configuration object
FieldTypeDescription

type

string

The IPAM address type. The value whereabouts is required.

range

string

An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses.

exclude

array

Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned.

Dynamic IP address assignment configuration example that uses Whereabouts

  1. {
  2. "ipam": {
  3. "type": "whereabouts",
  4. "range": "192.0.2.192/27",
  5. "exclude": [
  6. "192.0.2.192/30",
  7. "192.0.2.196/32"
  8. ]
  9. }
  10. }

Creating a Whereabouts reconciler daemon set

The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster using the Whereabouts IP Address Management (IPAM) solution. It ensures that each pods gets a unique IP address from the specified IP address range. It also handles IP address releases when pods are deleted or scaled down.

You can also use a NetworkAttachmentDefinition custom resource for dynamic IP address assignment.

The Whereabouts reconciler daemon set is automatically created when you configure an additional network through the Cluster Network Operator. It is not automatically created when you configure an additional network from a YAML manifest.

To trigger the deployment of the Whereabouts reconciler daemonset, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource file.

Use the following procedure to deploy the Whereabouts reconciler daemonset.

Procedure

  1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

    1. $ oc edit network.operator.openshift.io cluster
  2. Modify the additionalNetworks parameter in the CR to add the whereabouts-shim network attachment definition. For example:

    1. apiVersion: operator.openshift.io/v1
    2. kind: Network
    3. metadata:
    4. name: cluster
    5. spec:
    6. additionalNetworks:
    7. - name: whereabouts-shim
    8. namespace: default
    9. rawCNIConfig: |-
    10. {
    11. "name": "whereabouts-shim",
    12. "cniVersion": "0.3.1",
    13. "type": "bridge",
    14. "ipam": {
    15. "type": "whereabouts"
    16. }
    17. }
    18. type: Raw
  3. Save the file and exit the text editor.

  4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

    1. $ oc get all -n openshift-multus | grep whereabouts-reconciler

    Example output

    1. pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
    2. pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
    3. pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
    4. pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
    5. pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
    6. pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
    7. daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Creating a configuration for assignment of dual-stack IP addresses dynamically

Dual-stack IP address assignment can be configured with the ipRanges parameter for:

  • IPv4 addresses

  • IPv6 addresses

  • multiple IP address assignment

Procedure

  1. Set type to whereabouts.

  2. Use ipRanges to allocate IP addresses as shown in the following example:

    1. cniVersion: operator.openshift.io/v1
    2. kind: Network
    3. =metadata:
    4. name: cluster
    5. spec:
    6. additionalNetworks:
    7. - name: whereabouts-shim
    8. namespace: default
    9. type: Raw
    10. rawCNIConfig: |-
    11. {
    12. "name": "whereabouts-dual-stack",
    13. "cniVersion": "0.3.1,
    14. "type": "bridge",
    15. "ipam": {
    16. "type": "whereabouts",
    17. "ipRanges": [
    18. {"range": "192.168.10.0/24"},
    19. {"range": "2001:db8::/64"}
    20. ]
    21. }
    22. }
  3. Attach network to a pod. For more information, see “Adding a pod to an additional network”.

  4. Verify that all IP addresses are assigned.

  5. Run the following command to ensure the IP addresses are assigned as metadata.

    1. $ oc exec -it mypod -- ip a

Additional resources

Configuring SR-IOV additional network

You can configure an additional network that uses SR-IOV hardware by creating an SriovIBNetwork object. When you create an SriovIBNetwork object, the SR-IOV Network Operator automatically creates a NetworkAttachmentDefinition object.

Do not modify or delete an SriovIBNetwork object if it is attached to any pods in a running state.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a SriovIBNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

    1. apiVersion: sriovnetwork.openshift.io/v1
    2. kind: SriovIBNetwork
    3. metadata:
    4. name: attach1
    5. namespace: openshift-sriov-network-operator
    6. spec:
    7. resourceName: net1
    8. networkNamespace: project2
    9. ipam: |-
    10. {
    11. "type": "host-local",
    12. "subnet": "10.56.217.0/24",
    13. "rangeStart": "10.56.217.171",
    14. "rangeEnd": "10.56.217.181",
    15. "gateway": "10.56.217.1"
    16. }
  2. To create the object, enter the following command:

    1. $ oc create -f <name>.yaml

    where <name> specifies the name of the additional network.

  3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovIBNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovIBNetwork object.

    1. $ oc get net-attach-def -n <namespace>

Next steps

Additional resources