Converting a connected cluster to a disconnected cluster

There might be some scenarios where you need to convert your OKD cluster from a connected cluster to a disconnected cluster.

A disconnected cluster, also known as a restricted cluster, does not have an active connection to the internet. As such, you must mirror the contents of your registries and installation media. You can create this mirror registry on a host that can access both the internet and your closed network, or copy images to a device that you can move across network boundaries.

This topic describes the general process for converting an existing, connected cluster into a disconnected cluster.

The process for converting a cluster from connected to disconnected is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

About the mirror registry

You can mirror the images that are required for OKD installation and subsequent product updates to a mirror registry. These actions use the same process. The release image, which contains the description of the content, and the images it references are all mirrored. In addition, the Operator catalog source image and the images that it references must be mirrored for each Operator that you use. After you mirror the content, you configure each cluster to retrieve this content from your mirror registry.

The mirror registry can be any container registry that supports Docker v2-2. All major cloud provider registries, as well as Red Hat Quay, Artifactory, and others, have the necessary support. Using one of these registries ensures that OKD can verify the integrity of each image in disconnected environments.

The internal registry of the OKD cluster cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process.

The mirror registry must be reachable by every machine in the clusters that you provision. If the registry is unreachable installation, updating, or normal operations such as workload relocation might fail. For that reason, you must run mirror registries in a highly available way, and the mirror registries must at least match the production availability of your OKD clusters.

When you populate a mirror registry with OKD images, you can follow two scenarios. If you have a host that can access both the internet and your mirror registry, but not your cluster nodes, you can directly mirror the content from that machine. This process is referred to as connected mirroring. If you have no such host, you must mirror the images to a file system and then bring that host or removable media into your restricted environment. This process is referred to as disconnected mirroring.

For mirrored registries, to view the source of pulled images, you must review the Trying to access log entry in the CRI-O logs. Other methods to view the image pull source, such as using the crictl images command on a node, show the non-mirrored image name, even though the image is pulled from the mirrored location.

Prerequisites

Preparing the cluster for mirroring

Before disconnecting your cluster, you must mirror, or copy, the images to a mirror registry that is reachable by every node in your disconnected cluster. In order to mirror the images, you must prepare your cluster by:

  • Adding the mirror registry certificates to the list of trusted CAs on your host.

  • Creating a .dockerconfigjson file that contains your image pull secret, which is from the cloud.openshift.com token.

Procedure

  1. Configuring credentials that allow image mirroring:

    1. Add the CA certificate for the mirror registry, in the simple PEM or DER file formats, to the list of trusted CAs. For example:

      1. $ cp </path/to/cert.crt> /usr/share/pki/ca-trust-source/anchors/

      where

      </path/to/cert.crt>

      Specifies the path to the certificate on your local file system.

    2. Update the CA trust. For example, in Linux:

      1. $ update-ca-trust

    3. Extract the .dockerconfigjson file from the global pull secret:

      1. $ oc extract secret/pull-secret -n openshift-config --confirm --to=.

      Example output

      1. .dockerconfigjson

    4. Edit the .dockerconfigjson file to add your mirror registry and authentication credentials and save it as a new file:

      1. {"auths":{"<local_registry>": {"auth": "<credentials>","email": "you@example.com"}}},"<registry>:<port>/<namespace>/":{"auth":"<token>"}}}

      where:

      <local_registry>

      Specifies the registry domain name, and optionally the port, that your mirror registry uses to serve content.

      auth

      Specifies the base64-encoded user name and password for your mirror registry.

      <registry>:<port>/<namespace>

      Specifies the mirror registry details.

      <token>

      Specifies the base64-encoded username:password for your mirror registry.

      For example:

      1. $ {"auths":{"cloud.openshift.com":{"auth":"b3BlbnNoaWZ0Y3UjhGOVZPT0lOMEFaUjdPUzRGTA==","email":"user@example.com"},
      2. "quay.io":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGOVZPT0lOMEFaUGSTd4VGVGVUjdPUzRGTA==","email":"user@example.com"},
      3. "registry.connect.redhat.com"{"auth":"NTE3MTMwNDB8dWhjLTFEZlN3VHkxOSTd4VGVGVU1MdTpleUpoYkdjaUailA==","email":"user@example.com"},
      4. "registry.redhat.io":{"auth":"NTE3MTMwNDB8dWhjLTFEZlN3VH3BGSTd4VGVGVU1MdTpleUpoYkdjaU9fZw==","email":"user@example.com"},
      5. "registry.svc.ci.openshift.org":{"auth":"dXNlcjpyWjAwWVFjSEJiT2RKVW1pSmg4dW92dGp1SXRxQ3RGN1pwajJhN1ZXeTRV"},"my-registry:5000/my-namespace/":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}

Mirroring the images

After the cluster is properly configured, you can mirror the images from your external repositories to the mirror repository.

Procedure

  1. Mirror the Operator Lifecycle Manager (OLM) images:

    1. $ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v{product-version} <mirror_registry>:<port>/olm -a <reg_creds>

    where:

    product-version

    Specifies the tag that corresponds to the version of OKD to install, such as 4.8.

    mirror_registry

    Specifies the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry.

    reg_creds

    Specifies the location of your modified .dockerconfigjson file.

    For example:

    1. $ oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.8 mirror.registry.com:443/olm -a ./.dockerconfigjson  --index-filter-by-os='.*'

  2. Mirror the content for any other Red Hat-provided Operator:

    1. $ oc adm catalog mirror <index_image> <mirror_registry>:<port>/<namespace> -a <reg_creds>

    where:

    index_image

    Specifies the index image for the catalog you want to mirror. For example, this might be a pruned index image that you created previously, or one of the source index images for the default catalogs, such as {index-image-pullspec}.

    mirror_registry

    Specifies the target registry and namespace to mirror the Operator content to, where <namespace> is any existing namespace on the registry.

    reg_creds

    Optional: Specifies the location of your registry credentials file, if required.

    For example:

    1. $ oc adm catalog mirror registry.redhat.io/redhat/community-operator-index:v4.8 mirror.registry.com:443/olm -a ./.dockerconfigjson  --index-filter-by-os='.*'

  3. Mirror the OKD image repository:

    1. $ oc adm release mirror -a .dockerconfigjson --from=quay.io/openshift-release-dev/ocp-release:v<product-version>-<architecture> --to=<local_registry>/<local_repository> --to-release-image=<local_registry>/<local_repository>:v<product-version>-<architecture>

    where:

    product-version

    Specifies the tag that corresponds to the version of OKD to install, such as 4.8.15-x86_64.

    architecture

    Specifies the type of architecture for your server, such as x86_64.

    local_registry

    Specifies the registry domain name for your mirror repository.

    local_repository

    Specifies the name of the repository to create in your registry, such as ocp4/openshift4.

    For example:

    1. $ oc adm release mirror -a .dockerconfigjson --from=quay.io/openshift-release-dev/ocp-release:4.8.15-x86_64 --to=mirror.registry.com:443/ocp/release --to-release-image=mirror.registry.com:443/ocp/release:4.8.15-x86_64

    Example output

    1. info: Mirroring 109 images to mirror.registry.com/ocp/release ...
    2. mirror.registry.com:443/
    3.   ocp/release
    4.     manifests:
    5.       sha256:086224cadce475029065a0efc5244923f43fb9bb3bb47637e0aaf1f32b9cad47 -> 4.8.15-x86_64-thanos
    6.       sha256:0a214f12737cb1cfbec473cc301aa2c289d4837224c9603e99d1e90fc00328db -> 4.8.15-x86_64-kuryr-controller
    7.       sha256:0cf5fd36ac4b95f9de506623b902118a90ff17a07b663aad5d57c425ca44038c -> 4.8.15-x86_64-pod
    8.       sha256:0d1c356c26d6e5945a488ab2b050b75a8b838fc948a75c0fa13a9084974680cb -> 4.8.15-x86_64-kube-client-agent
    10. …..
    11. sha256:66e37d2532607e6c91eedf23b9600b4db904ce68e92b43c43d5b417ca6c8e63c mirror.registry.com:443/ocp/release:4.5.41-multus-admission-controller
    12. sha256:d36efdbf8d5b2cbc4dcdbd64297107d88a31ef6b0ec4a39695915c10db4973f1 mirror.registry.com:443/ocp/release:4.5.41-cluster-kube-scheduler-operator
    13. sha256:bd1baa5c8239b23ecdf76819ddb63cd1cd6091119fecdbf1a0db1fb3760321a2 mirror.registry.com:443/ocp/release:4.5.41-aws-machine-controllers
    14. info: Mirroring completed in 2.02s (0B/s)
    16. Success
    17. Update image:  mirror.registry.com:443/ocp/release:4.5.41-x86_64
    18. Mirror prefix: mirror.registry.com:443/ocp/release

  4. Mirror any other registries, as needed:

    1. $ oc image mirror <online_registry>/my/image:latest <mirror_registry>

Additional information

Configuring the cluster for the mirror registry

After creating and mirroring the images to the mirror registry, you must modify your cluster so that pods can pull images from the mirror registry.

You must:

  • Add the mirror registry credentials to the global pull secret.

  • Add the mirror registry server certificate to the cluster.

  • Create an ImageContentSourcePolicy custom resource (ICSP), which associates the mirror registry with the source registry.

    1. Add mirror registry credential to the cluster global pull-secret:

      1. $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> (1)
      1Provide the path to the new pull secret file.

      For example:

      1. $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=.mirrorsecretconfigjson

    2. Add the CA-signed mirror registry server certificate to the nodes in the cluster:

      1. Create a config map that includes the server certificate for the mirror registry

        1. $ oc create configmap <config_map_name> --from-file=<mirror_address_host>..<port>=$path/ca.crt -n openshift-config

        For example:

        1. S oc create configmap registry-config --from-file=mirror.registry.com..443=/root/certs/ca-chain.cert.pem -n openshift-config

      2. Use the config map to update the image.config.openshift.io/cluster custom resource (CR). OKD applies the changes to this CR to all nodes in the cluster:

        1. $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"<config_map_name>"}}}' --type=merge

        For example:

        1. $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

    3. Create an ICSP to redirect container pull requests from the online registries to the mirror registry:

      1. Create the ImageContentSourcePolicy custom resource:

        1. apiVersion: operator.openshift.io/v1alpha1
        2. kind: ImageContentSourcePolicy
        3. metadata:
        4.   name: mirror-ocp
        5. spec:
        6.   repositoryDigestMirrors:
        7.   - mirrors:
        8.     - mirror.registry.com:443/ocp/release (1)
        9.     source: quay.io/openshift-release-dev/ocp-release (2)
        10.   - mirrors:
        11.     - mirror.registry.com:443/ocp/release
        12.     source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
        1Specifies the name of the mirror image registry and repository.
        2Specifies the online registry and repository containing the content that is mirrored.

      2. Create the ICSP object:

        1. $ oc create -f registryrepomirror.yaml

        Example output

        1. imagecontentsourcepolicy.operator.openshift.io/mirror-ocp created

        OKD applies the changes to this CR to all nodes in the cluster.

    4. Verify that the credentials, CA, and ICSP for mirror registry were added:

      1. Log into a node:

        1. $ oc debug node/<node_name>

      2. Set /host as the root directory within the debug shell:

        1. sh-4.4# chroot /host

      3. Check the config.json file for the credentials:

        1. sh-4.4# cat /var/lib/kubelet/config.json

        Example output

        1. {"auths":{"brew.registry.redhat.io":{"xx=="},"brewregistry.stage.redhat.io":{"auth":"xxx=="},"mirror.registry.com:443":{"auth":"xx="}}} (1)
        1Ensure that the mirror registry and credentials are present.

      4. Change to the certs.d directory

        1. sh-4.4# cd /etc/docker/certs.d/

      5. List the certificates in the certs.d directory:

        1. sh-4.4# ls

        Example output

        1. image-registry.openshift-image-registry.svc.cluster.local:5000
        2. image-registry.openshift-image-registry.svc:5000
        3. mirror.registry.com:443 (1)
        1Ensure that the mirror registry is in the list.

      6. Check that the ICSP added the mirror registry to the registries.conf file:

        1. sh-4.4# cat /etc/containers/registries.conf

        Example output

        1. unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
        3. [[registry]]
        4.   prefix = ""
        5.   location = "quay.io/openshift-release-dev/ocp-release"
        6.   mirror-by-digest-only = true
        8.   [[registry.mirror]]
        9.     location = "mirror.registry.com:443/ocp/release"
        11. [[registry]]
        12.   prefix = ""
        13.   location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
        14.   mirror-by-digest-only = true
        16.   [[registry.mirror]]
        17.     location = "mirror.registry.com:443/ocp/release"

        The registry.mirror parameters indicate that the mirror registry is searched before the original registry.

      7. Exit the node.

        1. sh-4.4# exit

Ensure applications continue to work

Before disconnecting the cluster from the network, ensure that your cluster is working as expected and all of your applications are working as expected.

Procedure

Use the following commands to check the status of your cluster:

  • Ensure your pods are running:

    1. $ oc get pods --all-namespaces

    Example output

    1. NAMESPACE                                          NAME                                                          READY   STATUS      RESTARTS   AGE
    2. kube-system                                        apiserver-watcher-ci-ln-47ltxtb-f76d1-mrffg-master-0          1/1     Running     0          39m
    3. kube-system                                        apiserver-watcher-ci-ln-47ltxtb-f76d1-mrffg-master-1          1/1     Running     0          39m
    4. kube-system                                        apiserver-watcher-ci-ln-47ltxtb-f76d1-mrffg-master-2          1/1     Running     0          39m
    5. openshift-apiserver-operator                       openshift-apiserver-operator-79c7c646fd-5rvr5                 1/1     Running     3          45m
    6. openshift-apiserver                                apiserver-b944c4645-q694g                                     2/2     Running     0          29m
    7. openshift-apiserver                                apiserver-b944c4645-shdxb                                     2/2     Running     0          31m
    8. openshift-apiserver                                apiserver-b944c4645-x7rf2                                     2/2     Running     0          33m
    9.  ...

  • Ensure your nodes are in the READY status:

    1. $ oc get nodes

    Example output

    1. NAME                                       STATUS   ROLES    AGE   VERSION
    2. ci-ln-47ltxtb-f76d1-mrffg-master-0         Ready    master   42m   v1.21.1+a620f50
    3. ci-ln-47ltxtb-f76d1-mrffg-master-1         Ready    master   42m   v1.21.1+a620f50
    4. ci-ln-47ltxtb-f76d1-mrffg-master-2         Ready    master   42m   v1.21.1+a620f50
    5. ci-ln-47ltxtb-f76d1-mrffg-worker-a-gsxbz   Ready    worker   35m   v1.21.1+a620f50
    6. ci-ln-47ltxtb-f76d1-mrffg-worker-b-5qqdx   Ready    worker   35m   v1.21.1+a620f50
    7. ci-ln-47ltxtb-f76d1-mrffg-worker-c-rjkpq   Ready    worker   34m   v1.21.1+a620f50

Disconnect the cluster from the network

After mirroring all the required repositories and configuring your cluster to work as a disconnected cluster, you can disconnect the cluster from the network.

The Insights Operator is degraded when the cluster loses its Internet connection. You can avoid this problem by temporarily disabling the Insights Operator until you can restore it.

Restoring a degraded Insights Operator

Disconnecting the cluster from the network necessarily causes the cluster to lose the Internet connection. The Insights Operator becomes degraded because it requires access to Red Hat Insights.

This topic describes how to recover from a degraded Insights Operator.

Procedure

  1. Edit your .dockerconfigjson file to remove the cloud.openshift.com entry, for example:

    1. "cloud.openshift.com":{"auth":"<hash>","email":"user@example.com"}

  2. Save the file.

  3. Update the cluster secret with the edited .dockerconfigjson file:

    1. $ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=./.dockerconfigjson

  4. Verify that the Insights Operator is no longer degraded:

    1. $ oc get co insights

    Example output

    1. NAME       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    2. insights   4.5.41    True        False         False      3d

Restoring the network

If you want to reconnect a disconnected cluster and pull images from online registries, delete the cluster’s ImageContentSourcePolicy (ICSP) objects. Without the ICSP, pull requests to external registries are no longer redirected to the mirror registry.

Procedure

  1. View the ICSP objects in your cluster:

    1. $ oc get imagecontentsourcepolicy

    Example output

    1. NAME                 AGE
    2. mirror-ocp           6d20h
    3. ocp4-index-0         6d18h
    4. qe45-index-0         6d15h

  2. Delete all the ICSP objects you created when disconnecting your cluster:

    1. $ oc delete imagecontentsourcepolicy <icsp_name> <icsp_name> <icsp_name>

    For example:

    1. $ oc delete imagecontentsourcepolicy mirror-ocp ocp4-index-0 qe45-index-0

    Example output

    1. imagecontentsourcepolicy.operator.openshift.io "mirror-ocp" deleted
    2. imagecontentsourcepolicy.operator.openshift.io "ocp4-index-0" deleted
    3. imagecontentsourcepolicy.operator.openshift.io "qe45-index-0" deleted

  3. Wait for all the nodes to restart and return to the READY status and verify that the registries.conf file is pointing to the original registries and not the mirror registries:

    1. Log into a node:

      1. $ oc debug node/<node_name>

    2. Set /host as the root directory within the debug shell:

      1. sh-4.4# chroot /host

    3. Examine the registries.conf file:

      1. sh-4.4# cat /etc/containers/registries.conf

      Example output

      1. unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] (1)
      1The registry and registry.mirror entries created by the ICSPs you deleted are removed.