Migrating from the Kuryr network plugin to the OVN-Kubernetes network plugin

Migration from Kuryr to OVN-Kubernetes is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

As the administrator of a cluster that runs on OpenStack, you can migrate to the OVN-Kubernetes network plugin from the Kuryr SDN network plugin.

To learn more about OVN-Kubernetes, read About the OVN-Kubernetes network plugin.

Migration to the OVN-Kubernetes network provider

You can manually migrate a cluster that runs on OpenStack to the OVN-Kubernetes network provider.

Migration to OVN-Kubernetes is a one-way process. During migration, your cluster will be unreachable for a brief time.

Considerations when migrating to the OVN-Kubernetes network provider

Kubernetes namespaces are kept by Kuryr in separate OpenStack networking service (Neutron) subnets. Those subnets and the IP addresses that are assigned to individual pods are not preserved during the migration.

How the migration process works

The following table summarizes the migration process by relating the steps that you perform with the actions that your cluster and Operators take.

Table 1. The Kuryr to OVN-Kubernetes migration process
User-initiated stepsMigration activity

Set the migration field of the Network.operator.openshift.io custom resource (CR) named cluster to OVNKubernetes. Verify that the value of the migration field prints the null value before setting it to another value.

    Cluster Network Operator (CNO)

    Updates the status of the Network.config.openshift.io CR named cluster accordingly.

    Machine Config Operator (MCO)

    Deploys an update to the systemd configuration that is required by OVN-Kubernetes. By default, the MCO updates a single machine per pool at a time. As a result, large clusters have longer migration times.

Update the networkType field of the Network.config.openshift.io CR.

    CNO

    Performs the following actions:

    • Destroys the Kuryr control plane pods: Kuryr CNIs and the Kuryr controller.

    • Deploys the OVN-Kubernetes control plane pods.

    • Updates the Multus objects to reflect the new network plugin.

Reboot each node in the cluster.

    Cluster

    As nodes reboot, the cluster assigns IP addresses to pods on the OVN-Kubernetes cluster network.

Clean up remaining resources Kuryr controlled.

    Cluster

    Holds OpenStack resources that need to be freed, as well as OKD resources to configure.

Migrating to the OVN-Kubernetes network plugin

As a cluster administrator, you can change the network plugin for your cluster to OVN-Kubernetes.

During the migration, you must reboot every node in your cluster. Your cluster is unavailable and workloads might be interrupted. Perform the migration only if an interruption in service is acceptable.

Prerequisites

  • You installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

  • You have a recent backup of the etcd database is available.

  • You can manually reboot each node.

  • The cluster you plan to migrate is in a known good state, without any errors.

  • You installed the Python interpreter.

  • You installed the openstacksdk python package.

  • You installed the openstack CLI tool.

  • You have access to the underlying OpenStack cloud.

Procedure

  1. Back up the configuration for the cluster network by running the following command:

    1. $ oc get Network.config.openshift.io cluster -o yaml > cluster-kuryr.yaml
  2. To set the CLUSTERID variable, run the following command:

    1. $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')
  3. To prepare all the nodes for the migration, set the migration field on the Cluster Network Operator configuration object by running the following command:

    1. $ oc patch Network.operator.openshift.io cluster --type=merge \
    2. --patch '{"spec": {"migration": {"networkType": "OVNKubernetes"}}}'

    This step does not deploy OVN-Kubernetes immediately. Specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster. This prepares the cluster for the OVN-Kubernetes deployment.

  4. Optional: Customize the following settings for OVN-Kubernetes for your network infrastructure requirements:

    • Maximum transmission unit (MTU)

    • Geneve (Generic Network Virtualization Encapsulation) overlay network port

    • OVN-Kubernetes IPv4 internal subnet

    • OVN-Kubernetes IPv6 internal subnet

    To customize these settings, enter and customize the following command:

    1. $ oc patch Network.operator.openshift.io cluster --type=merge \
    2. --patch '{
    3. "spec":{
    4. "defaultNetwork":{
    5. "ovnKubernetesConfig":{
    6. "mtu":<mtu>,
    7. "genevePort":<port>,
    8. "v4InternalSubnet":"<ipv4_subnet>",
    9. "v6InternalSubnet":"<ipv6_subnet>"
    10. }}}}'

    where:

    mtu

    Specifies the MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

    port

    Specifies the UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by Kuryr. The default value for the VXLAN port is 4789.

    ipv4_subnet

    Specifies an IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

    ipv6_subnet

    Specifies an IPv6 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is fd98::/48.

    If you do not need to change the default value, omit the key from the patch.

    Example patch command to update mtu field

    1. $ oc patch Network.operator.openshift.io cluster --type=merge \
    2. --patch '{
    3. "spec":{
    4. "defaultNetwork":{
    5. "ovnKubernetesConfig":{
    6. "mtu":1200
    7. }}}}'
  5. Check the machine config pool status by entering the following command:

    1. $ oc get mcp

    While the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated before continuing.

    A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

    By default, the MCO updates one machine per pool at a time. Large clusters take more time to migrate than small clusters.

  6. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      1. $ oc describe node | egrep "hostname|machineconfig"

      Example output

      1. kubernetes.io/hostname=master-0
      2. machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b (2)
      3. machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b (3)
      4. machineconfiguration.openshift.io/reason:
      5. machineconfiguration.openshift.io/state: Done
    2. Review the output from the previous step. The following statements must be true:

      • The value of machineconfiguration.openshift.io/state field is Done.

      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    3. To confirm that the machine config is correct, enter the following command:

      1. $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where:

      <config_name>

      Specifies the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      Example output

      1. ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes
    4. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors:

      1. To list the pods, enter the following command:

        1. $ oc get pod -n openshift-machine-config-operator

        Example output

        1. NAME READY STATUS RESTARTS AGE
        2. machine-config-controller-75f756f89d-sjp8b 1/1 Running 0 37m
        3. machine-config-daemon-5cf4b 2/2 Running 0 43h
        4. machine-config-daemon-7wzcd 2/2 Running 0 43h
        5. machine-config-daemon-fc946 2/2 Running 0 43h
        6. machine-config-daemon-g2v28 2/2 Running 0 43h
        7. machine-config-daemon-gcl4f 2/2 Running 0 43h
        8. machine-config-daemon-l5tnv 2/2 Running 0 43h
        9. machine-config-operator-79d9c55d5-hth92 1/1 Running 0 37m
        10. machine-config-server-bsc8h 1/1 Running 0 43h
        11. machine-config-server-hklrm 1/1 Running 0 43h
        12. machine-config-server-k9rtx 1/1 Running 0 43h

        The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

        1. $ oc logs <pod> -n openshift-machine-config-operator

        where:

        <pod>

        Specifies the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

  1. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

    • To specify the network provider without changing the cluster network IP address block, enter the following command:

      1. $ oc patch Network.config.openshift.io cluster --type=merge \
      2. --patch '{"spec": {"networkType": "OVNKubernetes"}}'
    • To specify a different cluster network IP address block, enter the following command:

      1. $ oc patch Network.config.openshift.io cluster \
      2. --type='merge' --patch '{
      3. "spec": {
      4. "clusterNetwork": [
      5. {
      6. "cidr": "<cidr>",
      7. "hostPrefix": "<prefix>"
      8. }
      9. ]
      10. "networkType": "OVNKubernetes"
      11. }
      12. }'

      where:

      <cidr>

      Specifies a CIDR block.

      <prefix>

      Specifies a slice of the CIDR block that is apportioned to each node in your cluster.

      You cannot change the service network address block during the migration.

      You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

  2. To complete the migration, reboot each node in your cluster. For example, you can use a bash script similar to the following example. The script assumes that you can connect to each host by using ssh and that you have configured sudo to not prompt for a password:

    1. #!/bin/bash
    2. for ip in $(oc get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
    3. do
    4. echo "reboot node $ip"
    5. ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
    6. done

    If SSH access is not available, you can use the openstack command:

    1. $ for name in $(openstack server list name ${CLUSTERID}*” -f value -c Name); do openstack server reboot ${name}”; done

    Alternatively, you might be able to reboot each node through the management portal for your infrastructure provider. Otherwise, contact the appropriate authority to either gain access to the virtual machines through SSH or the management portal and OpenStack client.

Verification

  1. Confirm that the migration succeeded, and then remove the migration resources:

    1. To confirm that the network plugin is OVN-Kubernetes, enter the following command.

      1. $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

      The value of status.networkType must be OVNKubernetes.

    2. To confirm that the cluster nodes are in the Ready state, enter the following command:

      1. $ oc get nodes
    3. To confirm that your pods are not in an error state, enter the following command:

      1. $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

      If pods on a node are in an error state, reboot that node.

    4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

      1. $ oc get co

      The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

      Do not proceed if any of the previous verification steps indicate errors. You might encounter pods that have a Terminating state due to finalizers that are removed during clean up. They are not an error indication.

  2. If the migration completed and your cluster is in a good state, remove the migration configuration from the CNO configuration object by entering the following command:

    1. $ oc patch Network.operator.openshift.io cluster --type=merge \
    2. --patch '{"spec": {"migration": null}}'

Cleaning up resources after migration

After migration from the Kuryr network plugin to the OVN-Kubernetes network plugin, you must clean up the resources that Kuryr created previously.

The clean up process relies on a Python virtual environment to ensure that the package versions that you use support tags for Octavia objects. You do not need a virtual environment if you are certain that your environment uses at minimum:

  • The openstacksdk Python package version 0.54.0

  • The python-openstackclient Python package version 5.5.0

  • The python-octaviaclient Python package version 2.3.0

If you decide to use these particular versions, be sure to pull python-neutronclient prior to version 9.0.0, as it prevents you from accessing trunks.

Prerequisites

  • You installed the OKD CLI (oc).

  • You installed a Python interpreter.

  • You installed the openstacksdk Python package.

  • You installed the openstack CLI.

  • You have access to the underlying OpenStack cloud.

  • You can access the cluster as a user with the cluster-admin role.

Procedure

  1. Create a clean-up Python virtual environment:

    1. Create a temporary directory for your environment. For example:

      1. $ python3 -m venv /tmp/venv

      The virtual environment located in /tmp/venv directory is used in all clean up examples.

    2. Enter the virtual environment. For example:

      1. $ source /tmp/venv/bin/activate
    3. Upgrade the pip command in the virtual environment by running the following command:

      1. (venv) $ pip install --upgrade pip
    4. Install the required Python packages by running the following command:

      1. (venv) $ pip install openstacksdk==0.54.0 python-openstackclient==5.5.0 python-octaviaclient==2.3.0 'python-neutronclient<9.0.0'
  2. In your terminal, set variables to cluster and Kuryr identifiers by running the following commands:

    1. Set the cluster ID:

      1. (venv) $ CLUSTERID=$(oc get infrastructure.config.openshift.io cluster -o=jsonpath='{.status.infrastructureName}')
    2. Set the cluster tag:

      1. (venv) $ CLUSTERTAG="openshiftClusterID=${CLUSTERID}"
    3. Set the router ID:

      1. (venv) $ ROUTERID=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.routerId"|uniq)
  3. Create a Bash function that removes finalizers from specified resources by running the following command:

    1. (venv) $ function REMFIN {
    2. local resource=$1
    3. local finalizer=$2
    4. for res in $(oc get "${resource}" -A --template='{{range $i,$p := .items}}{{ $p.metadata.name }}|{{ $p.metadata.namespace }}{{"\n"}}{{end}}'); do
    5. name=${res%%|*}
    6. ns=${res##*|}
    7. yaml=$(oc get -n "${ns}" "${resource}" "${name}" -o yaml)
    8. if echo "${yaml}" | grep -q "${finalizer}"; then
    9. echo "${yaml}" | grep -v "${finalizer}" | oc replace -n "${ns}" "${resource}" "${name}" -f -
    10. fi
    11. done
    12. }

    The function takes two parameters: the first parameter is name of the resource, and the second parameter is the finalizer to remove. The named resource is removed from the cluster and its definition is replaced with copied data, excluding the specified finalizer.

  4. To remove Kuryr finalizers from services, enter the following command:

    1. (venv) $ REMFIN services kuryr.openstack.org/service-finalizer
  5. To remove the Kuryr service-subnet-gateway-ip service, enter the following command:

    1. (venv) $ if oc get -n openshift-kuryr service service-subnet-gateway-ip &>/dev/null; then
    2. oc -n openshift-kuryr delete service service-subnet-gateway-ip
    3. fi
  6. To remove all tagged OpenStack load balancers from Octavia, enter the following command:

    1. (venv) $ for lb in $(openstack loadbalancer list --tags "${CLUSTERTAG}" -f value -c id); do
    2. openstack loadbalancer delete --cascade "${lb}"
    3. done
  7. To remove Kuryr finalizers from all KuryrLoadBalancer CRs, enter the following command:

    1. (venv) $ REMFIN kuryrloadbalancers.openstack.org kuryr.openstack.org/kuryrloadbalancer-finalizers
  8. To remove the openshift-kuryr namespace, enter the following command:

    1. (venv) $ oc delete namespace openshift-kuryr
  9. To remove the Kuryr service subnet from the router, enter the following command:

    1. (venv) $ openstack router remove subnet "${ROUTERID}" "${CLUSTERID}-kuryr-service-subnet"
  10. To remove the Kuryr service network, enter the following command:

    1. (venv) $ openstack network delete "${CLUSTERID}-kuryr-service-network"
  11. To remove Kuryr finalizers from all pods, enter the following command:

    1. (venv) $ REMFIN pods kuryr.openstack.org/pod-finalizer
  12. To remove Kuryr finalizers from all KuryrPort CRs, enter the following command:

    1. (venv) $ REMFIN kuryrports.openstack.org kuryr.openstack.org/kuryrport-finalizer

    This command deletes the KuryrPort CRs.

  13. To remove Kuryr finalizers from network policies, enter the following command:

    1. (venv) $ REMFIN networkpolicy kuryr.openstack.org/networkpolicy-finalizer
  14. To remove Kuryr finalizers from remaining network policies, enter the following command:

    1. (venv) $ REMFIN kuryrnetworkpolicies.openstack.org kuryr.openstack.org/networkpolicy-finalizer
  15. To remove subports that Kuryr created from trunks, enter the following command:

    1. (venv) $ mapfile trunks < <(python -c "import openstack; n = openstack.connect().network; print('\n'.join([x.id for x in n.trunks(any_tags='$CLUSTERTAG')]))") && \
    2. i=0 && \
    3. for trunk in "${trunks[@]}"; do
    4. trunk=$(echo "$trunk"|tr -d '\n')
    5. i=$((i+1))
    6. echo "Processing trunk $trunk, ${i}/${#trunks[@]}."
    7. subports=()
    8. for subport in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x['port_id'] for x in n.get_trunk('$trunk').sub_ports if '$CLUSTERTAG' in n.get_port(x['port_id']).tags]))"); do
    9. subports+=("$subport");
    10. done
    11. args=()
    12. for sub in "${subports[@]}" ; do
    13. args+=("--subport $sub")
    14. done
    15. if [ ${#args[@]} -gt 0 ]; then
    16. openstack network trunk unset ${args[*]} "${trunk}"
    17. fi
    18. done
  16. To retrieve all networks and subnets from KuryrNetwork CRs and remove ports, router interfaces and the network itself, enter the following command:

    1. (venv) $ mapfile -t kuryrnetworks < <(oc get kuryrnetwork -A --template='{{range $i,$p := .items}}{{ $p.status.netId }}|{{ $p.status.subnetId }}{{"\n"}}{{end}}') && \
    2. i=0 && \
    3. for kn in "${kuryrnetworks[@]}"; do
    4. i=$((i+1))
    5. netID=${kn%%|*}
    6. subnetID=${kn##*|}
    7. echo "Processing network $netID, ${i}/${#kuryrnetworks[@]}"
    8. # Remove all ports from the network.
    9. for port in $(python -c "import openstack; n = openstack.connect().network; print(' '.join([x.id for x in n.ports(network_id='$netID') if x.device_owner != 'network:router_interface']))"); do
    10. ( openstack port delete "${port}" ) &
    11. # Only allow 20 jobs in parallel.
    12. if [[ $(jobs -r -p | wc -l) -ge 20 ]]; then
    13. wait -n
    14. fi
    15. done
    16. wait
    17. # Remove the subnet from the router.
    18. openstack router remove subnet "${ROUTERID}" "${subnetID}"
    19. # Remove the network.
    20. openstack network delete "${netID}"
    21. done
  17. To remove the Kuryr security group, enter the following command:

    1. (venv) $ openstack security group delete "${CLUSTERID}-kuryr-pods-security-group"
  18. To remove all tagged subnet pools, enter the following command:

    1. (venv) $ for subnetpool in $(openstack subnet pool list --tags "${CLUSTERTAG}" -f value -c ID); do
    2. openstack subnet pool delete "${subnetpool}"
    3. done
  19. To check that all of the networks based on KuryrNetwork CRs were removed, enter the following command:

    1. (venv) $ networks=$(oc get kuryrnetwork -A --no-headers -o custom-columns=":status.netId") && \
    2. for existingNet in $(openstack network list --tags "${CLUSTERTAG}" -f value -c ID); do
    3. if [[ $networks =~ $existingNet ]]; then
    4. echo "Network still exists: $existingNet"
    5. fi
    6. done

    If the command returns any existing networks, intestigate and remove them before you continue.

  20. To remove security groups that are related to network policy, enter the following command:

    1. (venv) $ for sgid in $(openstack security group list -f value -c ID -c Description | grep 'Kuryr-Kubernetes Network Policy' | cut -f 1 -d ' '); do
    2. openstack security group delete "${sgid}"
    3. done
  21. To remove finalizers from KuryrNetwork CRs, enter the following command:

    1. (venv) $ REMFIN kuryrnetworks.openstack.org kuryrnetwork.finalizers.kuryr.openstack.org
  22. To remove the Kuryr router, enter the following command:

    1. (venv) $ if python3 -c "import sys; import openstack; n = openstack.connect().network; r = n.get_router('$ROUTERID'); sys.exit(0) if r.description != 'Created By OpenShift Installer' else sys.exit(1)"; then
    2. openstack router delete "${ROUTERID}"
    3. fi

Additional resources