Controller Runtime Client API

Overview

The controller-runtime library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. The Operator SDK uses controller-runtime’s Client interface, which provides the interface for these operations.

controller-runtime defines several interfaces used for cluster interaction:

  • client.Client: implementers perform CRUD operations on a Kubernetes cluster.
  • manager.Manager: manages shared dependencies, such as Caches and Clients.
  • reconcile.Reconciler: compares provided state with actual cluster state and updates the cluster on finding state differences using a Client.

Clients are the focus of this document. A separate document will discuss Managers.

Client Usage

Default Client

The SDK relies on a manager.Manager to create a client.Client interface that performs Create, Update, Delete, Get, and List operations within a reconcile.Reconciler‘s Reconcile function. The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller’s Reconciler will be populated with the Manager’s Client which is a split-client.

controllers/<kind>_controller.go:

  1. import ctrl "sigs.k8s.io/controller-runtime"
  2. func (r *KindReconciler) SetupWithManager(mgr ctrl.Manager) error {
  3. return ctrl.NewControllerManagedBy(mgr).
  4. For(&cachev1alpha1.Kind{}).
  5. Owns(&appsv1.Deployment{}).
  6. Complete(r)
  7. }
  8. type KindReconciler struct {
  9. // Populated above from a manager.Manager.
  10. client.Client
  11. Log logr.Logger
  12. Scheme *runtime.Scheme
  13. }

A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.

Non-default Client

An operator developer may wish to create their own Client that serves read requests(Get List) from the API server instead of the cache, for example. controller-runtime provides a constructor for Clients:

  1. // New returns a new Client using the provided config and Options.
  2. func New(config *rest.Config, options client.Options) (client.Client, error)

client.Options allow the caller to specify how the new Client should communicate with the API server.

  1. // Options are creation options for a Client
  2. type Options struct {
  3. // Scheme, if provided, will be used to map go structs to GroupVersionKinds
  4. Scheme *runtime.Scheme
  5. // Mapper, if provided, will be used to map GroupVersionKinds to Resources
  6. Mapper meta.RESTMapper
  7. }

Example:

  1. import (
  2. "sigs.k8s.io/controller-runtime/pkg/client/config"
  3. "sigs.k8s.io/controller-runtime/pkg/client"
  4. )
  5. cfg, err := config.GetConfig()
  6. ...
  7. c, err := client.New(cfg, client.Options{})
  8. ...

Note: defaults are set by client.New when Options are empty. The default scheme will have the core Kubernetes resource types registered. The caller must set a scheme that has custom operator types registered for the new Client to recognize these types.

Creating a new Client is not usually necessary nor advised, as the default Client is sufficient for most use cases.

Reconcile and the Client API

A Reconciler implements the reconcile.Reconciler interface, which exposes the Reconcile method. Reconcilers are added to a corresponding Controller for a Kind; Reconcile is called in response to cluster or external Events, with a reconcile.Request object argument, to read and write cluster state by the Controller, and returns a reconcile.Result. SDK Reconcilers have access to a Client in order to make Kubernetes API calls.

  1. // KindReconciler reconciles a Kind object
  2. type KindReconciler struct {
  3. // client, initialized using mgr.Client() above, is a split client
  4. // that reads objects from the cache and writes to the apiserver
  5. client.Client
  6. Log logr.Logger
  7. // scheme defines methods for serializing and deserializing API objects,
  8. // a type registry for converting group, version, and kind information
  9. // to and from Go schemas, and mappings between Go schemas of different
  10. // versions. A scheme is the foundation for a versioned API and versioned
  11. // configuration over time.
  12. Scheme *runtime.Scheme
  13. }
  14. // Reconcile watches for Events and reconciles cluster state with desired
  15. // state defined in the method body.
  16. // The Controller will requeue the Request to be processed again if an error
  17. // is non-nil or Result.Requeue is true, otherwise upon completion it will
  18. // remove the work from the queue.
  19. func (r *KindReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error)

Reconcile is where Controller business logic lives, i.e. where Client API calls are made via KindReconciler.client. A client.Client implementer performs the following operations:

Get

  1. // Get retrieves an API object for a given object key from the Kubernetes cluster
  2. // and stores it in obj.
  3. func (c Client) Get(ctx context.Context, key ObjectKey, obj runtime.Object) error

Note: An ObjectKey is simply a client package alias for types.NamespacedName.

Example:

  1. import (
  2. "context"
  3. "github.com/example-org/app-operator/pkg/apis/cache/v1alpha1"
  4. ctrl "sigs.k8s.io/controller-runtime"
  5. )
  6. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  7. ...
  8. app := &v1alpha1.App{}
  9. ctx := context.TODO()
  10. err := r.Get(ctx, request.NamespacedName, app)
  11. ...
  12. }

List

  1. // List retrieves a list of objects for a given namespace and list options
  2. // and stores the list in obj.
  3. func (c Client) List(ctx context.Context, list runtime.Object, opts ...client.ListOption) error

A client.ListOption is an interface that sets client.ListOptions fields. A client.ListOption is created by using one of the provided implementations: MatchingLabels, MatchingFields, InNamespace.

Example:

  1. import (
  2. "context"
  3. "fmt"
  4. "k8s.io/api/core/v1"
  5. "sigs.k8s.io/controller-runtime/pkg/client"
  6. ctrl "sigs.k8s.io/controller-runtime"
  7. )
  8. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  9. ...
  10. // Return all pods in the request namespace with a label of `app=<name>`
  11. // and phase `Running`.
  12. podList := &v1.PodList{}
  13. opts := []client.ListOption{
  14. client.InNamespace(request.NamespacedName.Namespace),
  15. client.MatchingLabels{"app": request.NamespacedName.Name},
  16. client.MatchingFields{"status.phase": "Running"},
  17. }
  18. ctx := context.TODO()
  19. err := r.List(ctx, podList, opts...)
  20. ...
  21. }

Create

  1. // Create saves the object obj in the Kubernetes cluster.
  2. // Returns an error
  3. func (c Client) Create(ctx context.Context, obj runtime.Object, opts ...client.CreateOption) error

A client.CreateOption is an interface that sets client.CreateOptions fields. A client.CreateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

  1. import (
  2. "context"
  3. "k8s.io/api/apps/v1"
  4. ctrl "sigs.k8s.io/controller-runtime"
  5. )
  6. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  7. ...
  8. app := &v1.Deployment{ // Any cluster object you want to create.
  9. ...
  10. }
  11. ctx := context.TODO()
  12. err := r.Create(ctx, app)
  13. ...
  14. }

Update

  1. // Update updates the given obj in the Kubernetes cluster. obj must be a
  2. // struct pointer so that obj can be updated with the content returned
  3. // by the API server. Update does *not* update the resource's status
  4. // subresource
  5. func (c Client) Update(ctx context.Context, obj runtime.Object, opts ...client.UpdateOption) error

A client.UpdateOption is an interface that sets client.UpdateOptions fields. A client.UpdateOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

  1. import (
  2. "context"
  3. "k8s.io/api/apps/v1"
  4. ctrl "sigs.k8s.io/controller-runtime"
  5. )
  6. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  7. ...
  8. dep := &v1.Deployment{}
  9. err := r.Get(context.TODO(), request.NamespacedName, dep)
  10. ...
  11. ctx := context.TODO()
  12. dep.Spec.Selector.MatchLabels["is_running"] = "true"
  13. err := r.Update(ctx, dep)
  14. ...
  15. }

Patch

  1. // Patch patches the given obj in the Kubernetes cluster. obj must be a
  2. // struct pointer so that obj can be updated with the content returned by the Server.
  3. func (c Client) Patch(ctx context.Context, obj runtime.Object, patch client.Patch, opts ...client.UpdateOption) error

A client.PatchOption is an interface that sets client.PatchOptions fields. A client.PatchOption is created by using one of the provided implementations: DryRunAll, ForceOwnership. Generally these options are not needed.

Example:

  1. import (
  2. "context"
  3. "k8s.io/api/apps/v1"
  4. "sigs.k8s.io/controller-runtime/pkg/client"
  5. ctrl "sigs.k8s.io/controller-runtime"
  6. )
  7. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  8. ...
  9. dep := &v1.Deployment{}
  10. err := r.Get(context.TODO(), request.NamespacedName, dep)
  11. ...
  12. ctx := context.TODO()
  13. // A merge patch will preserve other fields modified at runtime.
  14. patch := client.MergeFrom(dep.DeepCopy())
  15. dep.Spec.Selector.MatchLabels["is_running"] = "true"
  16. err := r.Patch(ctx, dep, patch)
  17. ...
  18. }
Updating Status Subresource

When updating the status subresource from the client, the StatusWriter must be used. The status subresource is retrieved with Status() and updated with Update() or patched with Patch().

Update() takes variadic client.UpdateOption‘s, and Patch() takes variadic client.PatchOption‘s. See Client.Update() and Client.Patch() for more details. Generally these options are not needed.

Status
  1. // Status() returns a StatusWriter object that can be used to update the
  2. // object's status subresource
  3. func (c Client) Status() (client.StatusWriter, error)

Example:

  1. import (
  2. "context"
  3. cachev1alpha1 "github.com/example-inc/memcached-operator/pkg/apis/cache/v1alpha1"
  4. ctrl "sigs.k8s.io/controller-runtime"
  5. )
  6. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  7. ...
  8. ctx := context.TODO()
  9. mem := &cachev1alpha1.Memcached{}
  10. err := r.Get(ctx, request.NamespacedName, mem)
  11. ...
  12. // Update
  13. mem.Status.Nodes = []string{"pod1", "pod2"}
  14. err := r.Status().Update(ctx, mem)
  15. ...
  16. // Patch
  17. patch := client.MergeFrom(mem.DeepCopy())
  18. mem.Status.Nodes = []string{"pod1", "pod2", "pod3"}
  19. err := r.Status().Patch(ctx, mem, patch)
  20. ...
  21. }

Delete

  1. // Delete deletes the given obj from Kubernetes cluster.
  2. func (c Client) Delete(ctx context.Context, obj runtime.Object, opts ...client.DeleteOption) error

A client.DeleteOption is an interface that sets client.DeleteOptions fields. A client.DeleteOption is created by using one of the provided implementations: GracePeriodSeconds, Preconditions, PropagationPolicy.

Example:

  1. import (
  2. "context"
  3. "k8s.io/api/core/v1"
  4. "sigs.k8s.io/controller-runtime/pkg/client"
  5. ctrl "sigs.k8s.io/controller-runtime"
  6. )
  7. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  8. ...
  9. pod := &v1.Pod{}
  10. err := r.Get(context.TODO(), request.NamespacedName, pod)
  11. ...
  12. ctx := context.TODO()
  13. if pod.Status.Phase == v1.PodUnknown {
  14. // Delete the pod after 5 seconds.
  15. err := r.Delete(ctx, pod, client.GracePeriodSeconds(5))
  16. ...
  17. }
  18. ...
  19. }

DeleteAllOf

  1. // DeleteAllOf deletes all objects of the given type matching the given options.
  2. func (c Client) DeleteAllOf(ctx context.Context, obj runtime.Object, opts ...client.DeleteAllOfOption) error

A client.DeleteAllOfOption is an interface that sets client.DeleteAllOfOptions fields. A client.DeleteAllOfOption wraps a client.ListOption and client.DeleteOption.

Example:

  1. import (
  2. "context"
  3. "fmt"
  4. "k8s.io/api/core/v1"
  5. "sigs.k8s.io/controller-runtime/pkg/client"
  6. ctrl "sigs.k8s.io/controller-runtime"
  7. )
  8. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Request, error) {
  9. ...
  10. // Delete all pods in the request namespace with a label of `app=<name>`
  11. // and phase `Failed`.
  12. pod := &v1.Pod{}
  13. opts := []client.DeleteAllOfOption{
  14. client.InNamespace(request.NamespacedName.Namespace),
  15. client.MatchingLabels{"app", request.NamespacedName.Name},
  16. client.MatchingFields{"status.phase": "Failed"},
  17. client.GracePeriodSeconds(5),
  18. }
  19. ctx := context.TODO()
  20. err := r.DeleteAllOf(ctx, pod, opts...)
  21. ...
  22. }

Example usage

  1. import (
  2. "context"
  3. "reflect"
  4. appv1alpha1 "github.com/example-org/app-operator/api/v1alpha1"
  5. appsv1 "k8s.io/api/apps/v1"
  6. corev1 "k8s.io/api/core/v1"
  7. "k8s.io/apimachinery/pkg/api/errors"
  8. metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  9. "k8s.io/apimachinery/pkg/labels"
  10. "k8s.io/apimachinery/pkg/runtime"
  11. "k8s.io/apimachinery/pkg/types"
  12. "sigs.k8s.io/controller-runtime/pkg/client"
  13. "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
  14. ctrl "sigs.k8s.io/controller-runtime"
  15. )
  16. type AppReconciler struct {
  17. client.Client
  18. Log logr.Logger
  19. Scheme *runtime.Scheme
  20. }
  21. func (r *AppReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
  22. // Fetch the App instance.
  23. app := &appv1alpha1.App{}
  24. err := r.Get(context.TODO(), request.NamespacedName, app)
  25. if err != nil {
  26. if errors.IsNotFound(err) {
  27. return ctrl.Result{}, nil
  28. }
  29. return ctrl.Result{}, err
  30. }
  31. // Check if the deployment already exists, if not create a new deployment.
  32. found := &appsv1.Deployment{}
  33. err = r.Get(context.TODO(), types.NamespacedName{Name: app.Name, Namespace: app.Namespace}, found)
  34. if err != nil {
  35. if errors.IsNotFound(err) {
  36. // Define and create a new deployment.
  37. dep := r.deploymentForApp(app)
  38. if err = r.Create(context.TODO(), dep); err != nil {
  39. return ctrl.Result{}, err
  40. }
  41. return ctrl.Result{Requeue: true}, nil
  42. } else {
  43. return ctrl.Result{}, err
  44. }
  45. }
  46. // Ensure the deployment size is the same as the spec.
  47. size := app.Spec.Size
  48. if *found.Spec.Replicas != size {
  49. found.Spec.Replicas = &size
  50. if err = r.Update(context.TODO(), found); err != nil {
  51. return ctrl.Result{}, err
  52. }
  53. return ctrl.Result{Requeue: true}, nil
  54. }
  55. // Update the App status with the pod names.
  56. // List the pods for this app's deployment.
  57. podList := &corev1.PodList{}
  58. listOpts := []client.ListOption{
  59. client.InNamespace(app.Namespace),
  60. client.MatchingLabels(labelsForApp(app.Name)),
  61. }
  62. if err = r.List(context.TODO(), podList, listOpts...); err != nil {
  63. return ctrl.Result{}, err
  64. }
  65. // Update status.Nodes if needed.
  66. podNames := getPodNames(podList.Items)
  67. if !reflect.DeepEqual(podNames, app.Status.Nodes) {
  68. app.Status.Nodes = podNames
  69. if err := r.Status().Update(context.TODO(), app); err != nil {
  70. return ctrl.Result{}, err
  71. }
  72. }
  73. return ctrl.Request{}, nil
  74. }
  75. // deploymentForApp returns a app Deployment object.
  76. func (r *KindReconciler) deploymentForApp(m *appv1alpha1.App) *appsv1.Deployment {
  77. lbls := labelsForApp(m.Name)
  78. replicas := m.Spec.Size
  79. dep := &appsv1.Deployment{
  80. ObjectMeta: metav1.ObjectMeta{
  81. Name: m.Name,
  82. Namespace: m.Namespace,
  83. },
  84. Spec: appsv1.DeploymentSpec{
  85. Replicas: &replicas,
  86. Selector: &metav1.LabelSelector{
  87. MatchLabels: lbls,
  88. },
  89. Template: corev1.PodTemplateSpec{
  90. ObjectMeta: metav1.ObjectMeta{
  91. Labels: lbls,
  92. },
  93. Spec: corev1.PodSpec{
  94. Containers: []corev1.Container{{
  95. Image: "app:alpine",
  96. Name: "app",
  97. Command: []string{"app", "-a=64", "-b"},
  98. Ports: []corev1.ContainerPort{{
  99. ContainerPort: 10000,
  100. Name: "app",
  101. }},
  102. }},
  103. },
  104. },
  105. },
  106. }
  107. // Set App instance as the owner and controller.
  108. // NOTE: calling SetControllerReference, and setting owner references in
  109. // general, is important as it allows deleted objects to be garbage collected.
  110. controllerutil.SetControllerReference(m, dep, r.scheme)
  111. return dep
  112. }
  113. // labelsForApp creates a simple set of labels for App.
  114. func labelsForApp(name string) map[string]string {
  115. return map[string]string{"app_name": "app", "app_cr": name}
  116. }

Last modified July 17, 2020: [docs] Reorganize hugo doc site (#3382) (1248c1ca)