Creating Go-based Operators

Operator developers can take advantage of Go programming language support in the Operator SDK to build an example Go-based Operator for Memcached, a distributed key-value store, and manage its lifecycle.

Kubebuilder is embedded into the Operator SDK as the scaffolding solution for Go-based Operators.

Creating a Go-based Operator using the Operator SDK

The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. The SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code needed for many common management capabilities, such as metering or monitoring.

This procedure walks through an example of creating a simple Memcached Operator using tools and libraries provided by the SDK.

Prerequisites

  • Operator SDK v0.19.4 CLI installed on the development workstation

  • Operator Lifecycle Manager (OLM) installed on a Kubernetes-based cluster (v1.8 or above to support the apps/v1beta2 API group), for example OKD 4.6

  • Access to the cluster using an account with cluster-admin permissions

  • OpenShift CLI (oc) v4.6+ installed

Procedure

  1. Create an Operator project:

    1. Create a directory for the project:

      1. $ mkdir -p $HOME/projects/memcached-operator

    2. Change to the directory:

      1. $ cd $HOME/projects/memcached-operator

    3. Activate support for Go modules:

      1. $ export GO111MODULE=on

    4. Run the operator-sdk init command to initialize the project:

      1. $ operator-sdk init \
      2.     --domain=example.com \
      3.     --repo=github.com/example-inc/memcached-operator

      The operator-sdk init command uses the go.kubebuilder.io/v2 plug-in by default.

  2. Update your Operator to use supported images:

    1. In the project root-level Dockerfile, change the default runner image reference from:

      1. FROM gcr.io/distroless/static:nonroot

      to:

      1. FROM registry.access.redhat.com/ubi8/ubi-minimal:latest

    2. Depending on the Go project version, your Dockerfile might contain a USER 65532:65532 or USER nonroot:nonroot directive. In either case, remove the line, as it is not required by the supported runner image.

    3. In the config/default/manager_auth_proxy_patch.yaml file, change the image value from:

      1. gcr.io/kubebuilder/kube-rbac-proxy:<tag>

      to use the supported image:

      1. registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.6

  3. Update the test target in your Makefile to install dependencies required during later builds by replacing the following lines:

    Existing test target

    1. test: generate fmt vet manifests
    2.         go test ./... -coverprofile cover.out

    With the following lines:

    Updated test target

    1. ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
    2. test: manifests generate fmt vet ## Run tests.
    3.     mkdir -p ${ENVTEST_ASSETS_DIR}
    4.     test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.2/hack/setup-envtest.sh
    5.     source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

  4. Create a custom resource definition (CRD) API and controller:

    1. Run the following command to create an API with group cache, version v1, and kind Memcached:

      1. $ operator-sdk create api \
      2.     --group=cache \
      3.     --version=v1 \
      4.     --kind=Memcached

    2. When prompted, enter y for creating both the resource and controller:

      1. Create Resource [y/n]
      2. y
      3. Create Controller [y/n]
      4. y

      Example output

      1. Writing scaffold for you to edit...
      2. api/v1/memcached_types.go
      3. controllers/memcached_controller.go
      4. ...

      This process generates the Memcached resource API at api/v1/memcached_types.go and the controller at controllers/memcached_controller.go.

    3. Modify the Go type definitions at api/v1/memcached_types.go to have the following spec and status:

      1. // MemcachedSpec defines the desired state of Memcached
      2. type MemcachedSpec struct {
      3.     // +kubebuilder:validation:Minimum=0
      4.     // Size is the size of the memcached deployment
      5.     Size int32 `json:"size"`
      6. }
      8. // MemcachedStatus defines the observed state of Memcached
      9. type MemcachedStatus struct {
      10.     // Nodes are the names of the memcached pods
      11.     Nodes []string `json:"nodes"`
      12. }

    4. Add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest:

      1. // Memcached is the Schema for the memcacheds API
      2. // +kubebuilder:subresource:status (1)
      3. type Memcached struct {
      4.     metav1.TypeMeta   `json:",inline"`
      5.     metav1.ObjectMeta `json:"metadata,omitempty"`
      7.     Spec   MemcachedSpec   `json:"spec,omitempty"`
      8.     Status MemcachedStatus `json:"status,omitempty"`
      9. }
      1Add this line.

      This enables the controller to update the CR status without changing the rest of the CR object.

    5. Update the generated code for the resource type:

      1. $ make generate

      After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

      The above Makefile target invokes the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file. This ensures your API Go type definitions implement the runtime.Object interface that all Kind types must implement.

  5. Generate and update CRD manifests:

    1. $ make manifests

    This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.example.com_memcacheds.yaml file.

    1. Optional: Add custom validation to your CRD.

      OpenAPI v3.0 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached custom resource (CR) when it is created or updated.

      As an Operator author, you can use annotation-like, single-line comments called Kubebuilder markers to configure custom validations for your API. These markers must always have a +kubebuilder:validation prefix. For example, adding an enum-type specification can be done by adding the following marker:

      1. // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
      2. type Alias string

      Usage of markers in API code is discussed in the Kubebuilder Generating CRDs and Markers for Config/Code Generation documentation. A full list of OpenAPIv3 validation markers is also available in the Kubebuilder CRD Validation documentation.

      If you add any custom validations, run the following command to update the OpenAPI validation section for the CRD:

      1. $ make manifests

  6. After creating a new API and controller, you can implement the controller logic. For this example, replace the generated controller file controllers/memcached_controller.go with following example implementation:

    Example memcached_controller.go

    1. /*
    2. Licensed under the Apache License, Version 2.0 (the "License");
    3. you may not use this file except in compliance with the License.
    4. You may obtain a copy of the License at
    6.     http://www.apache.org/licenses/LICENSE-2.0
    8. Unless required by applicable law or agreed to in writing, software
    9. distributed under the License is distributed on an "AS IS" BASIS,
    10. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    11. See the License for the specific language governing permissions and
    12. limitations under the License.
    13. */
    15. package controllers
    17. import (
    18.     "context"
    19.     "reflect"
    21.     "github.com/go-logr/logr"
    22.     appsv1 "k8s.io/api/apps/v1"
    23.     corev1 "k8s.io/api/core/v1"
    24.     "k8s.io/apimachinery/pkg/api/errors"
    25.     metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    26.     "k8s.io/apimachinery/pkg/runtime"
    27.     "k8s.io/apimachinery/pkg/types"
    28.     ctrl "sigs.k8s.io/controller-runtime"
    29.     "sigs.k8s.io/controller-runtime/pkg/client"
    31.     cachev1 "github.com/example-inc/memcached-operator/api/v1"
    32. )
    34. // MemcachedReconciler reconciles a Memcached object
    35. type MemcachedReconciler struct {
    36.     client.Client
    37.     Log    logr.Logger
    38.     Scheme *runtime.Scheme
    39. }
    41. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
    42. // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
    43. // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
    44. // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;
    46. func (r *MemcachedReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    47.     ctx := context.Background()
    48.     log := r.Log.WithValues("memcached", req.NamespacedName)
    50.     // Fetch the Memcached instance
    51.     memcached := &cachev1.Memcached{}
    52.     err := r.Get(ctx, req.NamespacedName, memcached)
    53.     if err != nil {
    54.         if errors.IsNotFound(err) {
    55.             // Request object not found, could have been deleted after reconcile request.
    56.             // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
    57.             // Return and don't requeue
    58.             log.Info("Memcached resource not found. Ignoring since object must be deleted")
    59.             return ctrl.Result{}, nil
    60.         }
    61.         // Error reading the object - requeue the request.
    62.         log.Error(err, "Failed to get Memcached")
    63.         return ctrl.Result{}, err
    64.     }
    66.     // Check if the deployment already exists, if not create a new one
    67.     found := &appsv1.Deployment{}
    68.     err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
    69.     if err != nil && errors.IsNotFound(err) {
    70.         // Define a new deployment
    71.         dep := r.deploymentForMemcached(memcached)
    72.         log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
    73.         err = r.Create(ctx, dep)
    74.         if err != nil {
    75.             log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
    76.             return ctrl.Result{}, err
    77.         }
    78.         // Deployment created successfully - return and requeue
    79.         return ctrl.Result{Requeue: true}, nil
    80.     } else if err != nil {
    81.         log.Error(err, "Failed to get Deployment")
    82.         return ctrl.Result{}, err
    83.     }
    85.     // Ensure the deployment size is the same as the spec
    86.     size := memcached.Spec.Size
    87.     if *found.Spec.Replicas != size {
    88.         found.Spec.Replicas = &size
    89.         err = r.Update(ctx, found)
    90.         if err != nil {
    91.             log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
    92.             return ctrl.Result{}, err
    93.         }
    94.         // Spec updated - return and requeue
    95.         return ctrl.Result{Requeue: true}, nil
    96.     }
    98.     // Update the Memcached status with the pod names
    99.     // List the pods for this memcached's deployment
    100.     podList := &corev1.PodList{}
    101.     listOpts := []client.ListOption{
    102.         client.InNamespace(memcached.Namespace),
    103.         client.MatchingLabels(labelsForMemcached(memcached.Name)),
    104.     }
    105.     if err = r.List(ctx, podList, listOpts...); err != nil {
    106.         log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name)
    107.         return ctrl.Result{}, err
    108.     }
    109.     podNames := getPodNames(podList.Items)
    111.     // Update status.Nodes if needed
    112.     if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
    113.         memcached.Status.Nodes = podNames
    114.         err := r.Status().Update(ctx, memcached)
    115.         if err != nil {
    116.             log.Error(err, "Failed to update Memcached status")
    117.             return ctrl.Result{}, err
    118.         }
    119.     }
    121.     return ctrl.Result{}, nil
    122. }
    124. // deploymentForMemcached returns a memcached Deployment object
    125. func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1.Memcached) *appsv1.Deployment {
    126.     ls := labelsForMemcached(m.Name)
    127.     replicas := m.Spec.Size
    129.     dep := &appsv1.Deployment{
    130.         ObjectMeta: metav1.ObjectMeta{
    131.             Name:      m.Name,
    132.             Namespace: m.Namespace,
    133.         },
    134.         Spec: appsv1.DeploymentSpec{
    135.             Replicas: &replicas,
    136.             Selector: &metav1.LabelSelector{
    137.                 MatchLabels: ls,
    138.             },
    139.             Template: corev1.PodTemplateSpec{
    140.                 ObjectMeta: metav1.ObjectMeta{
    141.                     Labels: ls,
    142.                 },
    143.                 Spec: corev1.PodSpec{
    144.                     Containers: []corev1.Container{{
    145.                         Image:   "memcached:1.4.36-alpine",
    146.                         Name:    "memcached",
    147.                         Command: []string{"memcached", "-m=64", "-o", "modern", "-v"},
    148.                         Ports: []corev1.ContainerPort{{
    149.                             ContainerPort: 11211,
    150.                             Name:          "memcached",
    151.                         }},
    152.                     }},
    153.                 },
    154.             },
    155.         },
    156.     }
    157.     // Set Memcached instance as the owner and controller
    158.     ctrl.SetControllerReference(m, dep, r.Scheme)
    159.     return dep
    160. }
    162. // labelsForMemcached returns the labels for selecting the resources
    163. // belonging to the given memcached CR name.
    164. func labelsForMemcached(name string) map[string]string {
    165.     return map[string]string{"app": "memcached", "memcached_cr": name}
    166. }
    168. // getPodNames returns the pod names of the array of pods passed in
    169. func getPodNames(pods []corev1.Pod) []string {
    170.     var podNames []string
    171.     for _, pod := range pods {
    172.         podNames = append(podNames, pod.Name)
    173.     }
    174.     return podNames
    175. }
    177. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    178.     return ctrl.NewControllerManagedBy(mgr).
    179.         For(&cachev1.Memcached{}).
    180.         Owns(&appsv1.Deployment{}).
    181.         Complete(r)
    182. }

    The example controller runs the following reconciliation logic for each Memcached CR:

    • Create a Memcached deployment if it does not exist.

    • Ensure that the deployment size is the same as specified by the Memcached CR spec.

    • Update the Memcached CR status with the names of the memcached pods.

    The next two sub-steps inspect how the controller watches resources and how the reconcile loop is triggered. You can skip these steps to go directly to building and running the Operator.

    1. Inspect the controller implementation at the controllers/memcached_controller.go file to see how the controller watches resources.

      The SetupWithManager() function specifies how the controller is built to watch a CR and other resources that are owned and managed by that controller:

      SetupWithManager() function

      1. import (
      2.     ...
      3.     appsv1 "k8s.io/api/apps/v1"
      4.     ...
      5. )
      7. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
      8.     return ctrl.NewControllerManagedBy(mgr).
      9.         For(&cachev1.Memcached{}).
      10.         Owns(&appsv1.Deployment{}).
      11.         Complete(r)
      12. }

      NewControllerManagedBy() provides a controller builder that allows various controller configurations.

      For(&cachev1.Memcached{}) specifies the Memcached type as the primary resource to watch. For each Add, Update, or Delete event for a Memcached type, the reconcile loop is sent a reconcile Request argument, which consists of a namespace and name key, for that Memcached object.

      Owns(&appsv1.Deployment{}) specifies the Deployment type as the secondary resource to watch. For each Deployment type Add, Update, or Delete event, the event handler maps each event to a reconcile request for the owner of the deployment. In this case, the owner is the Memcached object for which the deployment was created.

    2. Every controller has a reconciler object with a Reconcile() method that implements the reconcile loop. The reconcile loop is passed the Request argument, which is a namespace and name key used to find the primary resource object, Memcached, from the cache:

      Reconcile loop

      1. import (
      2.     ctrl "sigs.k8s.io/controller-runtime"
      4.     cachev1 "github.com/example-inc/memcached-operator/api/v1"
      5.     ...
      6. )
      8. func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
      9.   // Lookup the Memcached instance for this reconcile request
      10.   memcached := &cachev1.Memcached{}
      11.   err := r.Get(ctx, req.NamespacedName, memcached)
      12.   ...
      13. }

      Based on the return value of the Reconcile() function, the reconcile Request might be requeued, and the loop might be triggered again:

      Requeue logic

      1. // Reconcile successful - don't requeue
      2. return reconcile.Result{}, nil
      3. // Reconcile failed due to error - requeue
      4. return reconcile.Result{}, err
      5. // Requeue for any reason other than error
      6. return reconcile.Result{Requeue: true}, nil

      You can set the Result.RequeueAfter to requeue the request after a grace period:

      Requeue after grace period

      1. import "time"
      3. // Reconcile for any reason other than an error after 5 seconds
      4. return ctrl.Result{RequeueAfter: time.Second*5}, nil

      You can return Result with RequeueAfter set to periodically reconcile a CR.

      For more on reconcilers, clients, and interacting with resource events, see the Controller Runtime Client API documentation.

Additional resources

Running the Operator

There are two ways you can use the Operator SDK CLI to build and run your Operator:

  • Run locally outside the cluster as a Go program.

  • Run as a deployment on the cluster.

Prerequisites

Running locally outside the cluster

You can run your Operator project as a Go program outside of the cluster. This method is useful for development purposes to speed up deployment and testing.

Procedure

  • Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config file and run the Operator as a Go program locally:

    1. $ make install run

    Example output

    1. ...
    2. 2021-01-10T21:09:29.016-0700    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": ":8080"}
    3. 2021-01-10T21:09:29.017-0700    INFO    setup    starting manager
    4. 2021-01-10T21:09:29.017-0700    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
    5. 2021-01-10T21:09:29.018-0700    INFO    controller-runtime.manager.controller.memcached    Starting EventSource    {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
    6. 2021-01-10T21:09:29.218-0700    INFO    controller-runtime.manager.controller.memcached    Starting Controller    {"reconciler group": "cache.example.com", "reconciler kind": "Memcached"}
    7. 2021-01-10T21:09:29.218-0700    INFO    controller-runtime.manager.controller.memcached    Starting workers    {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}

Running as a deployment

After creating your Go-based Operator project, you can build and run your Operator as a deployment inside a cluster.

Procedure

  1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

    1. Build the image:

      1. $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

    2. Push the image to a repository:

      1. $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

  2. Run the following command to deploy the Operator:

    1. $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

    By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

  3. Verify that the Operator is running:

    1. $ oc get deployment -n <project_name>-system

    Example output

    1. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
    2. <project_name>-controller-manager       1/1     1            1           8m

Creating a custom resource

After your Operator is installed, you can test it by creating a custom resource (CR) that is now provided on the cluster by the Operator.

Prerequisites

  • Example Memcached Operator, which provides the Memcached CR, installed on a cluster

Procedure

  1. Change to the namespace where your Operator is installed. For example, if you deployed the Operator using the make deploy command:

    1. $ oc project memcached-operator-system

  2. Edit the sample Memcached CR manifest at config/samples/cache_v1_memcached.yaml to contain the following specification:

    1. apiVersion: cache.example.com/v1
    2. kind: Memcached
    3. metadata:
    4.   name: memcached-sample
    5. ...
    6. spec:
    7. ...
    8.   size: 3

  3. Create the CR:

    1. $ oc apply -f config/samples/cache_v1_memcached.yaml

  4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

    1. $ oc get deployments

    Example output

    1. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
    2. memcached-operator-controller-manager   1/1     1            1           8m
    3. memcached-sample                        3/3     3            3           1m

  5. Check the pods and CR status to confirm the status is updated with the Memcached pod names.

    1. Check the pods:

      1. $ oc get pods

      Example output

      1. NAME                                  READY     STATUS    RESTARTS   AGE
      2. memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
      3. memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
      4. memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m

    2. Check the CR status:

      1. $ oc get memcached/memcached-sample -o yaml

      Example output

      1. apiVersion: cache.example.com/v1
      2. kind: Memcached
      3. metadata:
      4. ...
      5.   name: memcached-sample
      6. ...
      7. spec:
      8.   size: 3
      9. status:
      10.   nodes:
      11.   - memcached-sample-6fd7c98d8-7dqdr
      12.   - memcached-sample-6fd7c98d8-g5k7v
      13.   - memcached-sample-6fd7c98d8-m7vn7

  6. Update the deployment size.

    1. Update config/samples/cache_v1_memcached.yaml file to change the spec.size field in the Memcached CR from 3 to 5:

      1. $ oc patch memcached memcached-sample \
      2.     -p '{"spec":{"size": 5}}' \
      3.     --type=merge

    2. Confirm that the Operator changes the deployment size:

      1. $ oc get deployments

      Example output

      1. NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
      2. memcached-operator-controller-manager   1/1     1            1           10m
      3. memcached-sample                        5/5     5            5           3m

Additional resources

Getting involved

This guide provides an effective demonstration of the value of the Operator Framework for building and managing Operators, but this is much more left out in the interest of brevity. The Operator Framework and its components are open source, so visit each project individually and learn what else you can do:

github.com/operator-framework

If you want to discuss your experience, have questions, or want to get involved, join the Operator Framework mailing list.