Operator SDK tutorial for Go-based Operators

Operator developers can take advantage of Go programming language support in the Operator SDK to build an example Go-based Operator for Memcached, a distributed key-value store, and manage its lifecycle.

This process is accomplished using two centerpieces of the Operator Framework:

Operator SDK

The operator-sdk CLI tool and controller-runtime library API

Operator Lifecycle Manager (OLM)

Installation, upgrade, and role-based access control (RBAC) of Operators on a cluster

This tutorial goes into greater detail than Getting started with Operator SDK for Go-based Operators.

Prerequisites

Creating a project

Use the Operator SDK CLI to create a project called memcached-operator.

Procedure

  1. Create a directory for the project:

    1. $ mkdir -p $HOME/projects/memcached-operator
  2. Change to the directory:

    1. $ cd $HOME/projects/memcached-operator
  3. Activate support for Go modules:

    1. $ export GO111MODULE=on
  4. Run the operator-sdk init command to initialize the project:

    1. $ operator-sdk init \
    2. --domain=example.com \
    3. --repo=github.com/example-inc/memcached-operator

    The operator-sdk init command uses the Go plug-in by default.

    The operator-sdk init command generates a go.mod file to be used with Go modules. The --repo flag is required when creating a project outside of $GOPATH/src/, because generated files require a valid module path.

PROJECT file

Among the files generated by the operator-sdk init command is a Kubebuilder PROJECT file. Subsequent operator-sdk commands, as well as help output, that are run from the project root read this file and are aware that the project type is Go. For example:

  1. domain: example.com
  2. layout: go.kubebuilder.io/v3
  3. projectName: memcached-operator
  4. repo: github.com/example-inc/memcached-operator
  5. version: 3-alpha
  6. plugins:
  7. manifests.sdk.operatorframework.io/v2: {}
  8. scorecard.sdk.operatorframework.io/v2: {}

About the Manager

The main program for the Operator is the main.go file, which initializes and runs the Manager. The Manager automatically registers the Scheme for all custom resource (CR) API definitions and sets up and runs controllers and webhooks.

The Manager can restrict the namespace that all controllers watch for resources:

  1. mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: namespace})

By default, the Manager watches the namespace where the Operator runs. To watch all namespaces, you can leave the namespace option empty:

  1. mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: ""})

You can also use the MultiNamespacedCacheBuilder function to watch a specific set of namespaces:

  1. var namespaces []string (1)
  2. mgr, err := ctrl.NewManager(cfg, manager.Options{ (2)
  3. NewCache: cache.MultiNamespacedCacheBuilder(namespaces),
  4. })
1List of namespaces.
2Creates a Cmd struct to provide shared dependencies and start components.

About multi-group APIs

Before you create an API and controller, consider whether your Operator requires multiple API groups. This tutorial covers the default case of a single group API, but to change the layout of your project to support multi-group APIs, you can run the following command:

  1. $ operator-sdk edit --multigroup=true

This command updates the PROJECT file, which should look like the following example:

  1. domain: example.com
  2. layout: go.kubebuilder.io/v3
  3. multigroup: true
  4. ...

For multi-group projects, the API Go type files are created in the apis/<group>/<version>/ directory, and the controllers are created in the controllers/<group>/ directory. The Dockerfile is then updated accordingly.

Additional resource

Creating an API and controller

Use the Operator SDK CLI to create a custom resource definition (CRD) API and controller.

Procedure

  1. Run the following command to create an API with group cache, version, v1, and kind Memcached:

    1. $ operator-sdk create api \
    2. --group=cache \
    3. --version=v1 \
    4. --kind=Memcached
  2. When prompted, enter y for creating both the resource and controller:

    1. Create Resource [y/n]
    2. y
    3. Create Controller [y/n]
    4. y

    Example output

    1. Writing scaffold for you to edit...
    2. api/v1/memcached_types.go
    3. controllers/memcached_controller.go
    4. ...

This process generates the Memcached resource API at api/v1/memcached_types.go and the controller at controllers/memcached_controller.go.

Defining the API

Define the API for the Memcached custom resource (CR).

Procedure

  1. Modify the Go type definitions at api/v1/memcached_types.go to have the following spec and status:

    1. // MemcachedSpec defines the desired state of Memcached
    2. type MemcachedSpec struct {
    3. // +kubebuilder:validation:Minimum=0
    4. // Size is the size of the memcached deployment
    5. Size int32 `json:"size"`
    6. }
    7. // MemcachedStatus defines the observed state of Memcached
    8. type MemcachedStatus struct {
    9. // Nodes are the names of the memcached pods
    10. Nodes []string `json:"nodes"`
    11. }
  2. Add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest:

    1. // Memcached is the Schema for the memcacheds API
    2. // +kubebuilder:subresource:status (1)
    3. type Memcached struct {
    4. metav1.TypeMeta `json:",inline"`
    5. metav1.ObjectMeta `json:"metadata,omitempty"`
    6. Spec MemcachedSpec `json:"spec,omitempty"`
    7. Status MemcachedStatus `json:"status,omitempty"`
    8. }
    1Add this line.

    This enables the controller to update the CR status without changing the rest of the CR object.

  3. Update the generated code for the resource type:

    1. $ make generate

    After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

    The above Makefile target invokes the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file. This ensures your API Go type definitions implement the runtime.Object interface that all Kind types must implement.

Generating CRD manifests

After the API is defined with spec and status fields and custom resource definition (CRD) validation markers, you can generate CRD manifests.

Procedure

  • Run the following command to generate and update CRD manifests:

    1. $ make manifests

    This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.example.com_memcacheds.yaml file.

About OpenAPI validation

OpenAPIv3 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached custom resource (CR) when it is created or updated.

Markers, or annotations, are available to configure validations for your API. These markers always have a +kubebuilder:validation prefix.

Additional resources

Implementing the controller

After creating a new API and controller, you can implement the controller logic.

Procedure

  • For this example, replace the generated controller file controllers/memcached_controller.go with following example implementation:

    Example memcached_controller.go

    1. /*
    2. Copyright 2020.
    3. Licensed under the Apache License, Version 2.0 (the "License");
    4. you may not use this file except in compliance with the License.
    5. You may obtain a copy of the License at
    6. http://www.apache.org/licenses/LICENSE-2.0
    7. Unless required by applicable law or agreed to in writing, software
    8. distributed under the License is distributed on an "AS IS" BASIS,
    9. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    10. See the License for the specific language governing permissions and
    11. limitations under the License.
    12. */
    13. package controllers
    14. import (
    15. appsv1 "k8s.io/api/apps/v1"
    16. corev1 "k8s.io/api/core/v1"
    17. "k8s.io/apimachinery/pkg/api/errors"
    18. metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    19. "k8s.io/apimachinery/pkg/types"
    20. "reflect"
    21. "context"
    22. "github.com/go-logr/logr"
    23. "k8s.io/apimachinery/pkg/runtime"
    24. ctrl "sigs.k8s.io/controller-runtime"
    25. "sigs.k8s.io/controller-runtime/pkg/client"
    26. cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
    27. )
    28. // MemcachedReconciler reconciles a Memcached object
    29. type MemcachedReconciler struct {
    30. client.Client
    31. Log logr.Logger
    32. Scheme *runtime.Scheme
    33. }
    34. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
    35. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
    36. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
    37. // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
    38. // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
    39. // Reconcile is part of the main kubernetes reconciliation loop which aims to
    40. // move the current state of the cluster closer to the desired state.
    41. // TODO(user): Modify the Reconcile function to compare the state specified by
    42. // the Memcached object against the actual cluster state, and then
    43. // perform operations to make the cluster state reflect the state specified by
    44. // the user.
    45. //
    46. // For more details, check Reconcile and its Result here:
    47. // - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.7.0/pkg/reconcile
    48. func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    49. log := r.Log.WithValues("memcached", req.NamespacedName)
    50. // Fetch the Memcached instance
    51. memcached := &cachev1alpha1.Memcached{}
    52. err := r.Get(ctx, req.NamespacedName, memcached)
    53. if err != nil {
    54. if errors.IsNotFound(err) {
    55. // Request object not found, could have been deleted after reconcile request.
    56. // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
    57. // Return and don't requeue
    58. log.Info("Memcached resource not found. Ignoring since object must be deleted")
    59. return ctrl.Result{}, nil
    60. }
    61. // Error reading the object - requeue the request.
    62. log.Error(err, "Failed to get Memcached")
    63. return ctrl.Result{}, err
    64. }
    65. // Check if the deployment already exists, if not create a new one
    66. found := &appsv1.Deployment{}
    67. err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
    68. if err != nil && errors.IsNotFound(err) {
    69. // Define a new deployment
    70. dep := r.deploymentForMemcached(memcached)
    71. log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
    72. err = r.Create(ctx, dep)
    73. if err != nil {
    74. log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
    75. return ctrl.Result{}, err
    76. }
    77. // Deployment created successfully - return and requeue
    78. return ctrl.Result{Requeue: true}, nil
    79. } else if err != nil {
    80. log.Error(err, "Failed to get Deployment")
    81. return ctrl.Result{}, err
    82. }
    83. // Ensure the deployment size is the same as the spec
    84. size := memcached.Spec.Size
    85. if *found.Spec.Replicas != size {
    86. found.Spec.Replicas = &size
    87. err = r.Update(ctx, found)
    88. if err != nil {
    89. log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
    90. return ctrl.Result{}, err
    91. }
    92. // Spec updated - return and requeue
    93. return ctrl.Result{Requeue: true}, nil
    94. }
    95. // Update the Memcached status with the pod names
    96. // List the pods for this memcached's deployment
    97. podList := &corev1.PodList{}
    98. listOpts := []client.ListOption{
    99. client.InNamespace(memcached.Namespace),
    100. client.MatchingLabels(labelsForMemcached(memcached.Name)),
    101. }
    102. if err = r.List(ctx, podList, listOpts...); err != nil {
    103. log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
    104. return ctrl.Result{}, err
    105. }
    106. podNames := getPodNames(podList.Items)
    107. // Update status.Nodes if needed
    108. if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
    109. memcached.Status.Nodes = podNames
    110. err := r.Status().Update(ctx, memcached)
    111. if err != nil {
    112. log.Error(err, "Failed to update Memcached status")
    113. return ctrl.Result{}, err
    114. }
    115. }
    116. return ctrl.Result{}, nil
    117. }
    118. // deploymentForMemcached returns a memcached Deployment object
    119. func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment {
    120. ls := labelsForMemcached(m.Name)
    121. replicas := m.Spec.Size
    122. dep := &appsv1.Deployment{
    123. ObjectMeta: metav1.ObjectMeta{
    124. Name: m.Name,
    125. Namespace: m.Namespace,
    126. },
    127. Spec: appsv1.DeploymentSpec{
    128. Replicas: &replicas,
    129. Selector: &metav1.LabelSelector{
    130. MatchLabels: ls,
    131. },
    132. Template: corev1.PodTemplateSpec{
    133. ObjectMeta: metav1.ObjectMeta{
    134. Labels: ls,
    135. },
    136. Spec: corev1.PodSpec{
    137. Containers: []corev1.Container{{
    138. Image: "memcached:1.4.36-alpine",
    139. Name: "memcached",
    140. Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
    141. Ports: []corev1.ContainerPort{{
    142. ContainerPort: 11211,
    143. Name: "memcached",
    144. }},
    145. }},
    146. },
    147. },
    148. },
    149. }
    150. // Set Memcached instance as the owner and controller
    151. ctrl.SetControllerReference(m, dep, r.Scheme)
    152. return dep
    153. }
    154. // labelsForMemcached returns the labels for selecting the resources
    155. // belonging to the given memcached CR name.
    156. func labelsForMemcached(name string) map[string]string {
    157. return map[string]string{"app": "memcached", "memcached_cr": name}
    158. }
    159. // getPodNames returns the pod names of the array of pods passed in
    160. func getPodNames(pods []corev1.Pod) []string {
    161. var podNames []string
    162. for _, pod := range pods {
    163. podNames = append(podNames, pod.Name)
    164. }
    165. return podNames
    166. }
    167. // SetupWithManager sets up the controller with the Manager.
    168. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    169. return ctrl.NewControllerManagedBy(mgr).
    170. For(&cachev1alpha1.Memcached{}).
    171. Owns(&appsv1.Deployment{}).
    172. Complete(r)
    173. }

    The example controller runs the following reconciliation logic for each Memcached custom resource (CR):

    • Create a Memcached deployment if it does not exist.

    • Ensure that the deployment size is the same as specified by the Memcached CR spec.

    • Update the Memcached CR status with the names of the memcached pods.

The next subsections explain how the controller in the example implementation watches resources and how the reconcile loop is triggered. You can skip these subsections to go directly to Running the Operator.

Resources watched by the controller

The SetupWithManager() function in controllers/memcached_controller.go specifies how the controller is built to watch a CR and other resources that are owned and managed by that controller.

  1. import (
  2. ...
  3. appsv1 "k8s.io/api/apps/v1"
  4. ...
  5. )
  6. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
  7. return ctrl.NewControllerManagedBy(mgr).
  8. For(&cachev1.Memcached{}).
  9. Owns(&appsv1.Deployment{}).
  10. Complete(r)
  11. }

NewControllerManagedBy() provides a controller builder that allows various controller configurations.

For(&cachev1.Memcached{}) specifies the Memcached type as the primary resource to watch. For each Add, Update, or Delete event for a Memcached type, the reconcile loop is sent a reconcile Request argument, which consists of a namespace and name key, for that Memcached object.

Owns(&appsv1.Deployment{}) specifies the Deployment type as the secondary resource to watch. For each Deployment type Add, Update, or Delete event, the event handler maps each event to a reconcile request for the owner of the deployment. In this case, the owner is the Memcached object for which the deployment was created.

Controller configurations

You can initialize a controller by using many other useful configurations. For example:

  • Set the maximum number of concurrent reconciles for the controller by using the MaxConcurrentReconciles option, which defaults to 1:

    1. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    2. return ctrl.NewControllerManagedBy(mgr).
    3. For(&cachev1.Memcached{}).
    4. Owns(&appsv1.Deployment{}).
    5. WithOptions(controller.Options{
    6. MaxConcurrentReconciles: 2,
    7. }).
    8. Complete(r)
    9. }
  • Filter watch events using predicates.

  • Choose the type of EventHandler to change how a watch event translates to reconcile requests for the reconcile loop. For Operator relationships that are more complex than primary and secondary resources, you can use the EnqueueRequestsFromMapFunc handler to transform a watch event into an arbitrary set of reconcile requests.

For more details on these and other configurations, see the upstream Builder and Controller GoDocs.

Reconcile loop

Every controller has a reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument, which is a namespace and name key used to find the primary resource object, Memcached, from the cache:

  1. import (
  2. ctrl "sigs.k8s.io/controller-runtime"
  3. cachev1 "github.com/example-inc/memcached-operator/api/v1"
  4. ...
  5. )
  6. func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  7. // Lookup the Memcached instance for this reconcile request
  8. memcached := &cachev1.Memcached{}
  9. err := r.Get(ctx, req.NamespacedName, memcached)
  10. ...
  11. }

Based on the return values, result, and error, the request might be requeued and the reconcile loop might be triggered again:

  1. // Reconcile successful - don't requeue
  2. return ctrl.Result{}, nil
  3. // Reconcile failed due to error - requeue
  4. return ctrl.Result{}, err
  5. // Requeue for any reason other than an error
  6. return ctrl.Result{Requeue: true}, nil

You can set the Result.RequeueAfter to requeue the request after a grace period as well:

  1. import "time"
  2. // Reconcile for any reason other than an error after 5 seconds
  3. return ctrl.Result{RequeueAfter: time.Second*5}, nil

You can return Result with RequeueAfter set to periodically reconcile a CR.

For more on reconcilers, clients, and interacting with resource events, see the Controller Runtime Client API documentation.

Permissions and RBAC manifests

The controller requires certain RBAC permissions to interact with the resources it manages. These are specified using RBAC markers, such as the following:

  1. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
  2. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
  3. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
  4. // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
  5. // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
  6. func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  7. ...
  8. }

The ClusterRole object manifest at config/rbac/role.yaml is generated from the previous markers by using the controller-gen utility whenever the make manifests command is run.

Running the Operator

There are three ways you can use the Operator SDK CLI to build and run your Operator:

  • Run locally outside the cluster as a Go program.

  • Run as a deployment on the cluster.

  • Bundle your Operator and use Operator Lifecycle Manager (OLM) to deploy on the cluster.

Before running your Go-based Operator as either a deployment on OKD or as a bundle that uses OLM, ensure that your project has been updated to use supported images.

Running locally outside the cluster

You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.

Procedure

  • Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config file and run the Operator locally:

    1. $ make install run

    Example output

    1. ...
    2. 2021-01-10T21:09:29.016-0700 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
    3. 2021-01-10T21:09:29.017-0700 INFO setup starting manager
    4. 2021-01-10T21:09:29.017-0700 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
    5. 2021-01-10T21:09:29.018-0700 INFO controller-runtime.manager.controller.memcached Starting EventSource {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
    6. 2021-01-10T21:09:29.218-0700 INFO controller-runtime.manager.controller.memcached Starting Controller {"reconciler group": "cache.example.com", "reconciler kind": "Memcached"}
    7. 2021-01-10T21:09:29.218-0700 INFO controller-runtime.manager.controller.memcached Starting workers {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}

Running as a deployment on the cluster

You can run your Operator project as a deployment on your cluster.

Prerequisites

  • Prepared your Go-based Operator to run on OKD by updating the project to use supported images

Procedure

  1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

    1. Build the image:

      1. $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>
    2. Push the image to a repository:

      1. $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

  2. Run the following command to deploy the Operator:

    1. $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

    By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

  3. Verify that the Operator is running:

    1. $ oc get deployment -n <project_name>-system

    Example output

    1. NAME READY UP-TO-DATE AVAILABLE AGE
    2. <project_name>-controller-manager 1/1 1 1 8m

Bundling an Operator and deploying with Operator Lifecycle Manager

Bundling an Operator

The Operator bundle format is the default packaging method for Operator SDK and Operator Lifecycle Manager (OLM). You can get your Operator ready for use on OLM by using the Operator SDK to build and push your Operator project as a bundle image.

Prerequisites

  • Operator SDK CLI installed on a development workstation

  • OpenShift CLI (oc) v4.8+ installed

  • Operator project initialized by using the Operator SDK

  • If your Operator is Go-based, your project must be updated to use supported images for running on OKD

Procedure

  1. Run the following make commands in your Operator project directory to build and push your Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

    1. Build the image:

      1. $ make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
    2. Push the image to a repository:

      1. $ make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
  2. Update your Makefile by setting the IMG URL to your Operator image name and tag that you pushed:

    1. $ # Image URL to use all building/pushing image targets
    2. IMG ?= <registry>/<user>/<operator_image_name>:<tag>

    This value is used for subsequent operations.

  3. Create your Operator bundle manifest by running the make bundle command, which invokes several commands, including the Operator SDK generate bundle and bundle validate subcommands:

    1. $ make bundle

    Bundle manifests for an Operator describe how to display, create, and manage an application. The make bundle command creates the following files and directories in your Operator project:

    • A bundle manifests directory named bundle/manifests that contains a ClusterServiceVersion object

    • A bundle metadata directory named bundle/metadata

    • All custom resource definitions (CRDs) in a config/crd directory

    • A Dockerfile bundle.Dockerfile

    These files are then automatically validated by using operator-sdk bundle validate to ensure the on-disk bundle representation is correct.

  4. Build and push your bundle image by running the following commands. OLM consumes Operator bundles using an index image, which reference one or more bundle images.

    1. Build the bundle image. Set BUNDLE_IMG with the details for the registry, user namespace, and image tag where you intend to push the image:

      1. $ make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
    2. Push the bundle image:

      1. $ docker push <registry>/<user>/<bundle_image_name>:<tag>

Deploying an Operator with Operator Lifecycle Manager

Operator Lifecycle Manager (OLM) helps you to install, update, and manage the lifecycle of Operators and their associated services on a Kubernetes cluster. OLM is installed by default on OKD and runs as a Kubernetes extension so that you can use the web console and the OpenShift CLI (oc) for all Operator lifecycle management functions without any additional tools.

The Operator bundle format is the default packaging method for Operator SDK and OLM. You can use the Operator SDK to quickly run a bundle image on OLM to ensure that it runs properly.

Prerequisites

  • Operator SDK CLI installed on a development workstation

  • Operator bundle image built and pushed to a registry

  • OLM installed on a Kubernetes-based cluster (v1.16.0 or later if you use apiextensions.k8s.io/v1 CRDs, for example OKD 4.8)

  • Logged in to the cluster with oc using an account with cluster-admin permissions

  • If your Operator is Go-based, your project must be updated to use supported images for running on OKD

Procedure

  1. Check the status of OLM on your cluster by using the following Operator SDK command:

    1. $ operator-sdk olm status \
    2. --olm-namespace=openshift-operator-lifecycle-manager
  2. Run the Operator on your cluster by using the OLM integration in Operator SDK:

    1. $ operator-sdk run bundle \
    2. [-n <namespace>] \(1)
    3. <registry>/<user>/<bundle_image_name>:<tag>
    1By default, the command installs the Operator in the currently active project in your ~/.kube/config file. You can add the -n flag to set a different namespace scope for the installation.

    This command performs the following actions:

    • Create an index image referencing your bundle image. The index image is opaque and ephemeral, but accurately reflects how a bundle would be added to a catalog in production.

    • Create a catalog source that points to your new index image, which enables OperatorHub to discover your Operator.

    • Deploy your Operator to your cluster by creating an OperatorGroup, Subscription, InstallPlan, and all other required objects, including RBAC.

Creating a custom resource

After your Operator is installed, you can test it by creating a custom resource (CR) that is now provided on the cluster by the Operator.

Prerequisites

  • Example Memcached Operator, which provides the Memcached CR, installed on a cluster

Procedure

  1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

    1. $ oc project memcached-operator-system
  2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

    1. apiVersion: cache.example.com/v1
    2. kind: Memcached
    3. metadata:
    4. name: memcached-sample
    5. ...
    6. spec:
    7. ...
    8. size: 3
  3. Create the CR:

    1. $ oc apply -f config/samples/cache_v1_memcached.yaml
  4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

    1. $ oc get deployments

    Example output

    1. NAME READY UP-TO-DATE AVAILABLE AGE
    2. memcached-operator-controller-manager 1/1 1 1 8m
    3. memcached-sample 3/3 3 3 1m
  5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

    1. Check the pods:

      1. $ oc get pods

      Example output

      1. NAME READY STATUS RESTARTS AGE
      2. memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 1m
      3. memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 1m
      4. memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 1m
    2. Check the CR status:

      1. $ oc get memcached/memcached-sample -o yaml

      Example output

      1. apiVersion: cache.example.com/v1
      2. kind: Memcached
      3. metadata:
      4. ...
      5. name: memcached-sample
      6. ...
      7. spec:
      8. size: 3
      9. status:
      10. nodes:
      11. - memcached-sample-6fd7c98d8-7dqdr
      12. - memcached-sample-6fd7c98d8-g5k7v
      13. - memcached-sample-6fd7c98d8-m7vn7
  6. Update the deployment size.

    1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      1. $ oc patch memcached memcached-sample \
      2. -p '{"spec":{"size": 5}}' \
      3. --type=merge
    2. Confirm that the Operator changes the deployment size:

      1. $ oc get deployments

      Example output

      1. NAME READY UP-TO-DATE AVAILABLE AGE
      2. memcached-operator-controller-manager 1/1 1 1 10m
      3. memcached-sample 5/5 5 5 3m
  7. Clean up the resources that have been created as part of this tutorial.

    • If you used the make deploy command to test the Operator, run the following command:

      1. $ make undeploy
    • If you used the operator-sdk run bundle command to test the Operator, run the following command:

      1. $ operator-sdk cleanup <project_name>

Additional resources