Using Operator Lifecycle Manager on restricted networks

For OKD clusters that are installed on restricted networks, also known as disconnected clusters, Operator Lifecycle Manager (OLM) by default cannot access the Red Hat-provided OperatorHub sources hosted on remote registries because those remote sources require full internet connectivity.

However, as a cluster administrator you can still enable your cluster to use OLM in a restricted network if you have a workstation that has full internet access. The workstation, which requires full internet access to pull the remote OperatorHub content, is used to prepare local mirrors of the remote sources, and push the content to a mirror registry.

The mirror registry can be located on a bastion host, which requires connectivity to both your workstation and the disconnected cluster, or a completely disconnected, or airgapped, host, which requires removable media to physically move the mirrored content to the disconnected environment.

This guide describes the following process that is required to enable OLM in restricted networks:

  • Disable the default remote OperatorHub sources for OLM.

  • Use a workstation with full internet access to create and push local mirrors of the OperatorHub content to a mirror registry.

  • Configure OLM to install and manage Operators from local sources on the mirror registry instead of the default remote sources.

After enabling OLM in a restricted network, you can continue to use your unrestricted workstation to keep your local OperatorHub sources updated as newer versions of Operators are released.

While OLM can manage Operators from local sources, the ability for a given Operator to run successfully in a restricted network still depends on the Operator itself meeting the following criteria:

  • List any related images, or other container images that the Operator might require to perform their functions, in the relatedImages parameter of its ClusterServiceVersion (CSV) object.

  • Reference all specified images by a digest (SHA) and not by a tag.

You can search software on the Red Hat Ecosystem Catalog for a list of Red Hat Operators that support running in disconnected mode by filtering with the following selections:

Type

Containerized application

Deployment method

Operator

Infrastructure features

Disconnected

Additional resources

Prerequisites

  • Log in to your OKD cluster as a user with cluster-admin privileges.

If you are using OLM in a restricted network on IBM zSystems, you must have at least 12 GB allocated to the directory where you place your registry.

Disabling the default OperatorHub catalog sources

Operator catalogs that source content provided by Red Hat and community projects are configured for OperatorHub by default during an OKD installation. In a restricted network environment, you must disable the default catalogs as a cluster administrator. You can then configure OperatorHub to use local catalog sources.

Procedure

  • Disable the sources for the default catalogs by adding disableAllDefaultSources: true to the OperatorHub object:

    1. $ oc patch OperatorHub cluster --type json \
    2. -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Alternatively, you can use the web console to manage catalog sources. From the AdministrationCluster SettingsConfigurationOperatorHub page, click the Sources tab, where you can create, delete, disable, and enable individual sources.

Mirroring an Operator catalog

For instructions about mirroring Operator catalogs for use with disconnected clusters, see Installing → Mirroring images for a disconnected installation.

As of OKD 4.11, the default Red Hat-provided Operator catalog releases in the file-based catalog format. The default Red Hat-provided Operator catalogs for OKD 4.6 through 4.10 released in the deprecated SQLite database format.

The opm subcommands, flags, and functionality related to the SQLite database format are also deprecated and will be removed in a future release. The features are still supported and must be used for catalogs that use the deprecated SQLite database format.

Many of the opm subcommands and flags for working with the SQLite database format, such as opm index prune, do not work with the file-based catalog format. For more information about working with file-based catalogs, see Operator Framework packaging format, Managing custom catalogs, and Mirroring images for a disconnected installation using the oc-mirror plugin.

Adding a catalog source to a cluster

Adding a catalog source to an OKD cluster enables the discovery and installation of Operators for users. Cluster administrators can create a CatalogSource object that references an index image. OperatorHub uses catalog sources to populate the user interface.

Prerequisites

  • An index image built and pushed to a registry.

Procedure

  1. Create a CatalogSource object that references your index image. If you used the oc adm catalog mirror command to mirror your catalog to a target registry, you can use the generated catalogSource.yaml file in your manifests directory as a starting point.

    1. Modify the following to your specifications and save it as a catalogSource.yaml file:

      1. apiVersion: operators.coreos.com/v1alpha1
      2. kind: CatalogSource
      3. metadata:
      4. name: my-operator-catalog (1)
      5. namespace: olm (2)
      6. spec:
      7. sourceType: grpc
      8. grpcPodConfig:
      9. securityContextConfig: <security_mode> (3)
      10. image: <registry>/<namespace>/catalog:latest (4)
      11. displayName: My Operator Catalog
      12. publisher: <publisher_name> (5)
      13. updateStrategy:
      14. registryPoll: (6)
      15. interval: 30m
      1If you mirrored content to local files before uploading to a registry, remove any backslash (/) characters from the metadata.name field to avoid an “invalid resource name” error when you create the object.
      2If you want the catalog source to be available globally to users in all namespaces, specify the olm namespace. Otherwise, you can specify a different namespace for the catalog to be scoped and available only for that namespace.
      3Specify the value of legacy or restricted. If the field is not set, the default value is legacy. In a future OKD release, it is planned that the default value will be restricted. If your catalog cannot run with restricted permissions, it is recommended that you manually set this field to legacy.
      4Specify your index image. If you specify a tag after the image name, for example :latest, the catalog source pod uses an image pull policy of Always, meaning the pod always pulls the image prior to starting the container. If you specify a digest, for example @sha256:<id>, the image pull policy is IfNotPresent, meaning the pod pulls the image only if it does not already exist on the node.
      5Specify your name or an organization name publishing the catalog.
      6Catalog sources can automatically check for new versions to keep up to date.
    2. Use the file to create the CatalogSource object:

      1. $ oc apply -f catalogSource.yaml
  2. Verify the following resources are created successfully.

    1. Check the pods:

      1. $ oc get pods -n olm

      Example output

      1. NAME READY STATUS RESTARTS AGE
      2. my-operator-catalog-6njx6 1/1 Running 0 28s
      3. marketplace-operator-d9f549946-96sgr 1/1 Running 0 26h
    2. Check the catalog source:

      1. $ oc get catalogsource -n olm

      Example output

      1. NAME DISPLAY TYPE PUBLISHER AGE
      2. my-operator-catalog My Operator Catalog grpc 5s
    3. Check the package manifest:

      1. $ oc get packagemanifest -n olm

      Example output

      1. NAME CATALOG AGE
      2. jaeger-product My Operator Catalog 93s

You can now install the Operators from the OperatorHub page on your OKD web console.

Additional resources

Updating a SQLite-based index image

After configuring OperatorHub to use a catalog source that references a custom index image, cluster administrators can keep the available Operators on their cluster up to date by adding bundle images to the index image.

You can update an existing index image using the opm index add command. For restricted networks, the updated content must also be mirrored again to the cluster.

Prerequisites

  • opm

  • podman version 1.9.3+

  • An index image built and pushed to a registry.

  • An existing catalog source referencing the index image.

Procedure

  1. Update the existing index by adding bundle images:

    1. $ opm index add \
    2. --bundles <registry>/<namespace>/<new_bundle_image>@sha256:<digest> \(1)
    3. --from-index <registry>/<namespace>/<existing_index_image>:<existing_tag> \(2)
    4. --tag <registry>/<namespace>/<existing_index_image>:<updated_tag> \(3)
    5. --pull-tool podman (4)
    1The —bundles flag specifies a comma-separated list of additional bundle images to add to the index.
    2The —from-index flag specifies the previously pushed index.
    3The —tag flag specifies the image tag to apply to the updated index image.
    4The —pull-tool flag specifies the tool used to pull container images.

    where:

    <registry>

    Specifies the hostname of the registry, such as quay.io or mirror.example.com.

    <namespace>

    Specifies the namespace of the registry, such as ocs-dev or abc.

    <new_bundle_image>

    Specifies the new bundle image to add to the registry, such as ocs-operator.

    <digest>

    Specifies the SHA image ID, or digest, of the bundle image, such as c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41.

    <existing_index_image>

    Specifies the previously pushed image, such as abc-redhat-operator-index.

    <existing_tag>

    Specifies a previously pushed image tag, such as 4.13.

    <updated_tag>

    Specifies the image tag to apply to the updated index image, such as 4.13.1.

    Example command

    1. $ opm index add \
    2. --bundles quay.io/ocs-dev/ocs-operator@sha256:c7f11097a628f092d8bad148406aa0e0951094a03445fd4bc0775431ef683a41 \
    3. --from-index mirror.example.com/abc/abc-redhat-operator-index:4.13 \
    4. --tag mirror.example.com/abc/abc-redhat-operator-index:4.13.1 \
    5. --pull-tool podman
  2. Push the updated index image:

    1. $ podman push <registry>/<namespace>/<existing_index_image>:<updated_tag>
  3. Follow the steps in the Mirroring an Operator catalog procedure again to mirror the updated content. However, when you get to the step about creating the ImageContentSourcePolicy (ICSP) object, use the oc replace command instead of the oc create command. For example:

    1. $ oc replace -f ./manifests-catalog-<random_number>/imageContentSourcePolicy.yaml

    This change is required because the object already exists and must be updated.

    Normally, the oc apply command can be used to update existing objects that were previously created using oc apply. However, due to a known issue regarding the size of the metadata.annotations field in ICSP objects, the oc replace command must be used for this step currently.

  4. After Operator Lifecycle Manager (OLM) automatically polls the index image referenced in the catalog source at its regular interval, verify that the new packages are successfully added:

    1. $ oc get packagemanifests -n openshift-marketplace

Additional resources