Operator SDK tutorial for Hybrid Helm Operators

The standard Helm-based Operator support in the Operator SDK has limited functionality compared to the Go-based and Ansible-based Operator support that has reached the Auto Pilot capability (level V) in the Operator maturity model.

The Hybrid Helm Operator enhances the existing Helm-based support’s abilities through Go APIs. With this hybrid approach of Helm and Go, the Operator SDK enables Operator authors to use the following process:

  • Generate a default structure for, or scaffold, a Go API in the same project as Helm.

  • Configure the Helm reconciler in the main.go file of the project, through the libraries provided by the Hybrid Helm Operator.

The Hybrid Helm Operator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

This tutorial walks through the following process using the Hybrid Helm Operator:

  • Create a Memcached deployment through a Helm chart if it does not exist

  • Ensure that the deployment size is the same as specified by Memcached custom resource (CR) spec

  • Create a MemcachedBackup deployment by using the Go API

Prerequisites

  • Operator SDK CLI installed

  • OpenShift CLI (oc) v4.12+ installed

  • Logged into an OKD 4.12 cluster with oc with an account that has cluster-admin permissions

  • To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret

Additional resources

Creating a project

Use the Operator SDK CLI to create a project called memcached-operator.

Procedure

  1. Create a directory for the project:

    1. $ mkdir -p $HOME/github.com/example/memcached-operator
  2. Change to the directory:

    1. $ cd $HOME/github.com/example/memcached-operator
  3. Run the operator-sdk init command to initialize the project. Use a domain of example.com so that all API groups are <group>.example.com:

    1. $ operator-sdk init \
    2. --plugins=hybrid.helm.sdk.operatorframework.io \
    3. --project-version="3" \
    4. --domain example.com \
    5. --repo=github.com/example/memcached-operator

    The init command generates the RBAC rules in the config/rbac/role.yaml file based on the resources that would be deployed by the chart’s default manifests. Verify that the rules generated in the config/rbac/role.yaml file meet your Operator’s permission requirements.

Additional resources

  • This procedure creates a project structure that is compatible with both Helm and Go APIs. To learn more about the project directory structure, see Project layout.

Creating a Helm API

Use the Operator SDK CLI to create a Helm API.

Procedure

  • Run the following command to create a Helm API with group cache, version v1, and kind Memcached:

    1. $ operator-sdk create api \
    2. --plugins helm.sdk.operatorframework.io/v1 \
    3. --group cache \
    4. --version v1 \
    5. --kind Memcached

This procedure also configures your Operator project to watch the Memcached resource with API version v1 and scaffolds a boilerplate Helm chart. Instead of creating the project from the boilerplate Helm chart scaffolded by the Operator SDK, you can alternatively use an existing chart from your local file system or remote chart repository.

For more details and examples for creating Helm API based on existing or new charts, run the following command:

  1. $ operator-sdk create api plugins helm.sdk.operatorframework.io/v1 help

Additional resources

Operator logic for the Helm API

By default, your scaffolded Operator project watches Memcached resource events as shown in the watches.yaml file and executes Helm releases using the specified chart.

Example watches.yaml file

  1. # Use the 'create api' subcommand to add watches to this file.
  2. - group: cache.my.domain
  3. version: v1
  4. kind: Memcached
  5. chart: helm-charts/memcached
  6. #+kubebuilder:scaffold:watch

Additional resources

Custom Helm reconciler configurations using provided library APIs

A disadvantage of existing Helm-based Operators is the inability to configure the Helm reconciler, because it is abstracted from users. For a Helm-based Operator to reach the Seamless Upgrades capability (level II and later) that reuses an already existing Helm chart, a hybrid between the Go and Helm Operator types adds value.

The APIs provided in the helm-operator-plugins library allow Operator authors to make the following configurations:

  • Customize value mapping based on cluster state

  • Execute code in specific events by configuring the reconciler’s event recorder

  • Customize the reconciler’s logger

  • Setup Install, Upgrade, and Uninstall annotations to enable Helm’s actions to be configured based on the annotations found in custom resources watched by the reconciler

  • Configure the reconciler to run with Pre and Post hooks

The above configurations to the reconciler can be done in the main.go file:

Details

Example main.go file

  1. // Operator's main.go
  2. // With the help of helpers provided in the library, the reconciler can be
  3. // configured here before starting the controller with this reconciler.
  4. reconciler := reconciler.New(
  5. reconciler.WithChart(*chart),
  6. reconciler.WithGroupVersionKind(gvk),
  7. )
  8. if err := reconciler.SetupWithManager(mgr); err != nil {
  9. panic(fmt.Sprintf("unable to create reconciler: %s", err))
  10. }

Creating a Go API

Use the Operator SDK CLI to create a Go API.

Procedure

  1. Run the following command to create a Go API with group cache, version v1, and kind MemcachedBackup:

    1. $ operator-sdk create api \
    2. --group=cache \
    3. --version v1 \
    4. --kind MemcachedBackup \
    5. --resource \
    6. --controller \
    7. --plugins=go/v3
  2. When prompted, enter y for creating both resource and controller:

    1. $ Create Resource [y/n]
    2. y
    3. Create Controller [y/n]
    4. y

This procedure generates the MemcachedBackup resource API at api/v1/memcachedbackup_types.go and the controller at controllers/memcachedbackup_controller.go.

Defining the API

Define the API for the MemcachedBackup custom resource (CR).

Represent this Go API by defining the MemcachedBackup type, which will have a MemcachedBackupSpec.Size field to set the quantity of Memcached backup instances (CRs) to be deployed, and a MemcachedBackupStatus.Nodes field to store a CR’s pod names.

The Node field is used to illustrate an example of a Status field.

Procedure

  1. Define the API for the MemcachedBackup CR by modifying the Go type definitions in the api/v1/memcachedbackup_types.go file to have the following spec and status:

    Example api/v1/memcachedbackup_types.go file

    1. // MemcachedBackupSpec defines the desired state of MemcachedBackup
    2. type MemcachedBackupSpec struct {
    3. // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    4. // Important: Run "make" to regenerate code after modifying this file
    5. //+kubebuilder:validation:Minimum=0
    6. // Size is the size of the memcached deployment
    7. Size int32 `json:"size"`
    8. }
    9. // MemcachedBackupStatus defines the observed state of MemcachedBackup
    10. type MemcachedBackupStatus struct {
    11. // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    12. // Important: Run "make" to regenerate code after modifying this file
    13. // Nodes are the names of the memcached pods
    14. Nodes []string `json:"nodes"`
    15. }
  2. Update the generated code for the resource type:

    1. $ make generate

    After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type.

  3. After the API is defined with spec and status fields and CRD validation markers, generate and update the CRD manifests:

    1. $ make manifests

This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.my.domain_memcachedbackups.yaml file.

Controller implementation

The controller in this tutorial performs the following actions:

  • Create a Memcached deployment if it does not exist.

  • Ensure that the deployment size is the same as specified by the Memcached CR spec.

  • Update the Memcached CR status with the names of the memcached pods.

For a detailed explanation on how to configure the controller to perform the above mentioned actions, see Implementing the controller in the Operator SDK tutorial for standard Go-based Operators.

Differences in main.go

For standard Go-based Operators and the Hybrid Helm Operator, the main.go file handles the scaffolding the initialization and running of the Manager program for the Go API. For the Hybrid Helm Operator, however, the main.go file also exposes the logic for loading the watches.yaml file and configuring the Helm reconciler.

Example main.go file

  1. ...
  2. for _, w := range ws {
  3. // Register controller with the factory
  4. reconcilePeriod := defaultReconcilePeriod
  5. if w.ReconcilePeriod != nil {
  6. reconcilePeriod = w.ReconcilePeriod.Duration
  7. }
  8. maxConcurrentReconciles := defaultMaxConcurrentReconciles
  9. if w.MaxConcurrentReconciles != nil {
  10. maxConcurrentReconciles = *w.MaxConcurrentReconciles
  11. }
  12. r, err := reconciler.New(
  13. reconciler.WithChart(*w.Chart),
  14. reconciler.WithGroupVersionKind(w.GroupVersionKind),
  15. reconciler.WithOverrideValues(w.OverrideValues),
  16. reconciler.SkipDependentWatches(w.WatchDependentResources != nil && !*w.WatchDependentResources),
  17. reconciler.WithMaxConcurrentReconciles(maxConcurrentReconciles),
  18. reconciler.WithReconcilePeriod(reconcilePeriod),
  19. reconciler.WithInstallAnnotations(annotation.DefaultInstallAnnotations...),
  20. reconciler.WithUpgradeAnnotations(annotation.DefaultUpgradeAnnotations...),
  21. reconciler.WithUninstallAnnotations(annotation.DefaultUninstallAnnotations...),
  22. )
  23. ...

The manager is initialized with both Helm and Go reconcilers:

Example Helm and Go reconcilers

  1. ...
  2. // Setup manager with Go API
  3. if err = (&controllers.MemcachedBackupReconciler{
  4. Client: mgr.GetClient(),
  5. Scheme: mgr.GetScheme(),
  6. }).SetupWithManager(mgr); err != nil {
  7. setupLog.Error(err, "unable to create controller", "controller", "MemcachedBackup")
  8. os.Exit(1)
  9. }
  10. ...
  11. // Setup manager with Helm API
  12. for _, w := range ws {
  13. ...
  14. if err := r.SetupWithManager(mgr); err != nil {
  15. setupLog.Error(err, "unable to create controller", "controller", "Helm")
  16. os.Exit(1)
  17. }
  18. setupLog.Info("configured watch", "gvk", w.GroupVersionKind, "chartPath", w.ChartPath, "maxConcurrentReconciles", maxConcurrentReconciles, "reconcilePeriod", reconcilePeriod)
  19. }
  20. // Start the manager
  21. if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
  22. setupLog.Error(err, "problem running manager")
  23. os.Exit(1)
  24. }

Permissions and RBAC manifests

The controller requires certain role-based access control (RBAC) permissions to interact with the resources it manages. For the Go API, these are specified with RBAC markers, as shown in the Operator SDK tutorial for standard Go-based Operators.

For the Helm API, the permissions are scaffolded by default in roles.yaml. Currently, however, due to a known issue when the Go API is scaffolded, the permissions for the Helm API are overwritten. As a result of this issue, ensure that the permissions defined in roles.yaml match your requirements.

The following is an example role.yaml for a Memcached Operator:

Example Helm and Go reconcilers

  1. ---
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5. name: manager-role
  6. rules:
  7. - apiGroups:
  8. - ""
  9. resources:
  10. - namespaces
  11. verbs:
  12. - get
  13. - apiGroups:
  14. - apps
  15. resources:
  16. - deployments
  17. - daemonsets
  18. - replicasets
  19. - statefulsets
  20. verbs:
  21. - create
  22. - delete
  23. - get
  24. - list
  25. - patch
  26. - update
  27. - watch
  28. - apiGroups:
  29. - cache.my.domain
  30. resources:
  31. - memcachedbackups
  32. verbs:
  33. - create
  34. - delete
  35. - get
  36. - list
  37. - patch
  38. - update
  39. - watch
  40. - apiGroups:
  41. - cache.my.domain
  42. resources:
  43. - memcachedbackups/finalizers
  44. verbs:
  45. - create
  46. - delete
  47. - get
  48. - list
  49. - patch
  50. - update
  51. - watch
  52. - apiGroups:
  53. - ""
  54. resources:
  55. - pods
  56. - services
  57. - services/finalizers
  58. - endpoints
  59. - persistentvolumeclaims
  60. - events
  61. - configmaps
  62. - secrets
  63. - serviceaccounts
  64. verbs:
  65. - create
  66. - delete
  67. - get
  68. - list
  69. - patch
  70. - update
  71. - watch
  72. - apiGroups:
  73. - cache.my.domain
  74. resources:
  75. - memcachedbackups/status
  76. verbs:
  77. - get
  78. - patch
  79. - update
  80. - apiGroups:
  81. - policy
  82. resources:
  83. - events
  84. - poddisruptionbudgets
  85. verbs:
  86. - create
  87. - delete
  88. - get
  89. - list
  90. - patch
  91. - update
  92. - watch
  93. - apiGroups:
  94. - cache.my.domain
  95. resources:
  96. - memcacheds
  97. - memcacheds/status
  98. - memcacheds/finalizers
  99. verbs:
  100. - create
  101. - delete
  102. - get
  103. - list
  104. - patch
  105. - update
  106. - watch

Additional resources

Running locally outside the cluster

You can run your Operator project as a Go program outside of the cluster. This is useful for development purposes to speed up deployment and testing.

Procedure

  • Run the following command to install the custom resource definitions (CRDs) in the cluster configured in your ~/.kube/config file and run the Operator locally:

    1. $ make install run

Running as a deployment on the cluster

You can run your Operator project as a deployment on your cluster.

Procedure

  1. Run the following make commands to build and push the Operator image. Modify the IMG argument in the following steps to reference a repository that you have access to. You can obtain an account for storing containers at repository sites such as Quay.io.

    1. Build the image:

      1. $ make docker-build IMG=<registry>/<user>/<image_name>:<tag>

      The Dockerfile generated by the SDK for the Operator explicitly references GOARCH=amd64 for go build. This can be amended to GOARCH=$TARGETARCH for non-AMD64 architectures. Docker will automatically set the environment variable to the value specified by –platform. With Buildah, the –build-arg will need to be used for the purpose. For more information, see Multiple Architectures.

    2. Push the image to a repository:

      1. $ make docker-push IMG=<registry>/<user>/<image_name>:<tag>

      The name and tag of the image, for example IMG=<registry>/<user>/<image_name>:<tag>, in both the commands can also be set in your Makefile. Modify the IMG ?= controller:latest value to set your default image name.

  2. Run the following command to deploy the Operator:

    1. $ make deploy IMG=<registry>/<user>/<image_name>:<tag>

    By default, this command creates a namespace with the name of your Operator project in the form <project_name>-system and is used for the deployment. This command also installs the RBAC manifests from config/rbac.

  3. Run the following command to verify that the Operator is running:

    1. $ oc get deployment -n <project_name>-system

    Example output

    1. NAME READY UP-TO-DATE AVAILABLE AGE
    2. <project_name>-controller-manager 1/1 1 1 8m

Creating custom resources

After your Operator is installed, you can test it by creating custom resources (CRs) that are now provided on the cluster by the Operator.

Procedure

  1. Change to the namespace where your Operator is installed:

    1. $ oc project <project_name>-system
  2. Update the sample Memcached CR manifest at the config/samples/cache_v1_memcached.yaml file by updating the replicaCount field to 3:

    Example config/samples/cache_v1_memcached.yaml file

    1. apiVersion: cache.my.domain/v1
    2. kind: Memcached
    3. metadata:
    4. name: memcached-sample
    5. spec:
    6. # Default values copied from <project_dir>/helm-charts/memcached/values.yaml
    7. affinity: {}
    8. autoscaling:
    9. enabled: false
    10. maxReplicas: 100
    11. minReplicas: 1
    12. targetCPUUtilizationPercentage: 80
    13. fullnameOverride: ""
    14. image:
    15. pullPolicy: IfNotPresent
    16. repository: nginx
    17. tag: ""
    18. imagePullSecrets: []
    19. ingress:
    20. annotations: {}
    21. className: ""
    22. enabled: false
    23. hosts:
    24. - host: chart-example.local
    25. paths:
    26. - path: /
    27. pathType: ImplementationSpecific
    28. tls: []
    29. nameOverride: ""
    30. nodeSelector: {}
    31. podAnnotations: {}
    32. podSecurityContext: {}
    33. replicaCount: 3
    34. resources: {}
    35. securityContext: {}
    36. service:
    37. port: 80
    38. type: ClusterIP
    39. serviceAccount:
    40. annotations: {}
    41. create: true
    42. name: ""
    43. tolerations: []
  3. Create the Memcached CR:

    1. $ oc apply -f config/samples/cache_v1_memcached.yaml
  4. Ensure that the Memcached Operator creates the deployment for the sample CR with the correct size:

    1. $ oc get pods

    Example output

    1. NAME READY STATUS RESTARTS AGE
    2. memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 18m
    3. memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 18m
    4. memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 18m
  5. Update the sample MemcachedBackup CR manifest at the config/samples/cache_v1_memcachedbackup.yaml file by updating the size to 2:

    Example config/samples/cache_v1_memcachedbackup.yaml file

    1. apiVersion: cache.my.domain/v1
    2. kind: MemcachedBackup
    3. metadata:
    4. name: memcachedbackup-sample
    5. spec:
    6. size: 2
  6. Create the MemcachedBackup CR:

    1. $ oc apply -f config/samples/cache_v1_memcachedbackup.yaml
  7. Ensure that the count of memcachedbackup pods is the same as specified in the CR:

    1. $ oc get pods

    Example output

    1. NAME READY STATUS RESTARTS AGE
    2. memcachedbackup-sample-8649699989-4bbzg 1/1 Running 0 22m
    3. memcachedbackup-sample-8649699989-mq6mx 1/1 Running 0 22m
  8. You can update the spec in each of the above CRs, and then apply them again. The controller reconciles again and ensures that the size of the pods is as specified in the spec of the respective CRs.

  9. Clean up the resources that have been created as part of this tutorial:

    1. Delete the Memcached resource:

      1. $ oc delete -f config/samples/cache_v1_memcached.yaml
    2. Delete the MemcachedBackup resource:

      1. $ oc delete -f config/samples/cache_v1_memcachedbackup.yaml
    3. If you used the make deploy command to test the Operator, run the following command:

      1. $ make undeploy

Project layout

The Hybrid Helm Operator scaffolding is customized to be compatible with both Helm and Go APIs.

File/foldersPurpose

Dockerfile

Instructions used by a container engine to build your Operator image with the make docker-build command.

Makefile

Build file with helper targets to help you work with your project.

PROJECT

YAML file containing metadata information for the Operator. Represents the project’s configuration and is used to track useful information for the CLI and plugins.

bin/

Contains useful binaries such as the manager which is used to run your project locally and the kustomize utility used for the project configuration.

config/

Contains configuration files, including all Kustomize manifests, to launch your Operator project on a cluster. Plugins might use it to provide functionality. For example, for the Operator SDK to help create your Operator bundle, the CLI looks up the CRDs and CRs which are scaffolded in this directory.

    config/crd/

    Contains custom resource definitions (CRDs).

    config/default/

    Contains a Kustomize base for launching the controller in a standard configuration.

    config/manager/

    Contains the manifests to launch your Operator project as pods on the cluster.

    config/manifests/

    Contains the base to generate your OLM manifests in the bundle/ directory.

    config/prometheus/

    Contains the manifests required to enable project to serve metrics to Prometheus such as the ServiceMonitor resource.

    config/scorecard/

    Contains the manifests required to allow you test your project with the scorecard tool.

    config/rbac/

    Contains the RBAC permissions required to run your project.

    config/samples/

    Contains samples for custom resources.

api/

Contains the Go API definition.

controllers/

Contains the controllers for the Go API.

hack/

Contains utility files, such as the file used to scaffold the license header for your project files.

main.go

Main program of the Operator. Instantiates a new manager that registers all custom resource definitions (CRDs) in the apis/ directory and starts all controllers in the controllers/ directory.

helm-charts/

Contains the Helm charts which can be specified using the create api command with the Helm plugin.

watches.yaml

Contains group/version/kind (GVK) and Helm chart location. Used to configure the Helm watches.