Cloud-Native Devops Best Practices(2) - GitOps + OpenKruise CloneSet

What is GitOps？

GitOps is an approach to continuous delivery. Its core idea is to store the declarative infrastructure and applications of an application in a Git repository.

With Git at the core of the delivery pipeline, every developer can submit Pull Requests and use Git to accelerate and simplify application deployment and maintenance tasks for Kubernetes. By using a simple tool like Git, developers can more efficiently focus on creating new features rather than operations-related tasks (e.g., application installation, configuration, migration, etc.).

argo-cd

GitOps + OpenKruise CloneSet Practice

Requirements

  • Install Kubernetes Cluster, Since v1.0.0 (alpha/beta), OpenKruise requires Kubernetes version >= 1.16.
  • Install Tekton, Reference Official Documents, Tekton is a Google open source Kubernetes native framework for creating continuous integration and continuous deployment/delivery (CI/CD) systems.
  • Install Argo-cd, Reference Official Documents, Argo-cd is a declarative GitOps continuous delivery tool for Kubernetes.

Install OpenKruise（Enable: TemplateNoDefaults）

Openkruise installed by default will inject the default value of pod / PVC template, which will conflict with the sync judgment logic of Argo CD. Therefore, when installing openkruise, you need to open gates TemplateNoDefaults, as follows:

  1. # Firstly add openkruise charts repository if you haven't do this.
  2. $ helm repo add openkruise https://openkruise.github.io/charts/
  4. # [Optional]
  5. $ helm repo update
  7. # Install the latest version.
  8. $ helm install kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"
  10. # Those that have been installed need to be upgraded
  11. $ helm upgrade kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"

CloneSet Deploy Stateless Application

CloneSet is the ability provided by OpenKruise to efficiently manage stateless applications, it is similar to the official workload: Deployment, but offers many enhancements such as InPlace Update, Batch Release, etc. Please refer to the documentation CloneSet. This article provides a helloworld http service demo, It contains Helm Charts, and the cloneSet configuration is shown below:

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: CloneSet
  3. metadata:
  4.   name: helloworld-server
  5.   labels:
  6.     app: helloworld-server
  7. spec:
  8.   updateStrategy:
  9.     # CloneSet will try to in-place update Pod instead of recreating them if possible
  10.     type: InPlaceIfPossible
  11.     # Batch release, currently updating only one Pod
  12.     partition: 1
  13.   replicas: 2
  14.   selector:
  15.     matchLabels:
  16.       app: helloworld-server
  17.   template:
  18.     metadata:
  19.       labels:
  20.         app: helloworld-server
  21.     spec:
  22.       containers:
  23.       - name: helloworld
  24.         image: "openkruise/kruise:hello_world-d92ae174b"

Argo-cd CloneSet Health Check

Configure CloneSet Argo-cd Custom CRD Health Checks, With this configuration argo-cd is able to perform a healthy check of the CloneSet, such as whether the CloneSet is published and whether the Pods are ready, as follows：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   labels:
  5.     app.kubernetes.io/name: argocd-cm
  6.     app.kubernetes.io/part-of: argocd
  7.   name: argocd-cm
  8.   namespace: argocd
  9. data:
  10.   resource.customizations.health.apps.kruise.io_CloneSet: |
  11.     hs = {}
  12.     -- if paused
  13.     if obj.spec.updateStrategy.paused then
  14.       hs.status = "Suspended"
  15.       hs.message = "CloneSet is Suspended"
  16.       return hs
  17.     end
  19.     -- check cloneSet status
  20.     if obj.status ~= nil then
  21.       if obj.status.observedGeneration < obj.metadata.generation then
  22.         hs.status = "Progressing"
  23.         hs.message = "Waiting for rollout to finish: observed cloneSet generation less then desired generation"
  24.         return hs
  25.       end
  27.       if obj.status.updatedReplicas < obj.spec.replicas then
  28.         hs.status = "Progressing"
  29.         hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  30.         return hs
  31.       end
  33.       if obj.status.updatedReadyReplicas < obj.status.updatedReplicas then
  34.         hs.status = "Progressing"
  35.         hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  36.         return hs
  37.       end
  39.       hs.status = "Healthy"
  40.       return hs
  41.     end
  43.     -- if status == nil
  44.     hs.status = "Progressing"
  45.     hs.message = "Waiting for cloneSet"
  46.     return hs

OpenKruise internal CRD resources in addition to CloneSet, others such as: Advanced StatefulSet, SidecarSet, etc. can be similar to the above way to achieve Custom Resource Health.

Tekton Pipeline + Argo-cd

Argo-CD together with Tekton Pipeline is a popular DevOps practice and integrate well with CI process. Such practice requires storing the Argo-cd admin secret in K8S Secret CRD (method of obtaining secret), which in turn can be used in Tekton Pipeline, as follows:

  1. apiVersion: v1
  2. data:
  3.   # argo-cd admin secret
  4.   username: xxxxx
  5.   password: xxxxx
  6.   server: xxxxx
  7. kind: Secret
  8. metadata:
  9.   name: argosecret
  10. ---
  11. apiVersion: tekton.dev/v1beta1
  12. kind: Task
  13. metadata:
  14.   labels:
  15.     app: helloworld
  16.   name: helloworld-argocd
  17. spec:
  18.   params:
  19.   - name: gitrepositoryurl
  20.     type: string
  21.   - name: branch
  22.     type: string
  23.   - name: short_sha
  24.     type: string
  25.   - name: docker_repo
  26.     type: string
  27.   - name: app_name
  28.     type: string
  29.   - name: app_ns
  30.     type: string
  31.   - name: k8s_server
  32.     type: string
  33.   steps:
  34.   - name: argocd-deploy
  35.     image: argoproj/argocd:latest
  36.     command:
  37.     - sh
  38.     args:
  39.     - '-ce'
  40.     - >
  41.       set -e
  43.       echo "upgrade app $(params.app_name)"; username=`cat /var/secret/username`; password=`cat /var/secret/password`; server=`cat /var/secret/server`;
  45.       argocd login ${server} --insecure --username ${username} --password ${password}
  47.       argocd app create $(params.app_name) --upsert --repo $(params.gitrepositoryurl) --path $(params.app_name)/charts --dest-namespace $(params.app_ns) --dest-server $(params.k8s_server) --revision $(params.branch) --helm-set image.repository=$(params.docker_repo) --helm-set image.tag=$(params.branch)-$(params.short_sha) --helm-set installation.namespace=$(params.app_ns)
  49.       argocd app list; argocd app sync $(params.app_name)
  51.       argocd app wait $(params.app_name) --health
  52.     volumeMounts:
  53.     - name: argocd-secret
  54.       mountPath: "/var/secret"
  55.   volumes:
  56.   - name: argocd-secret
  57.     secret:
  58.       secretName: argosecret
  59. ---
  60. apiVersion: tekton.dev/v1beta1
  61. kind: Pipeline
  62. metadata:
  63.   name: helloworld-pipeline
  64. spec:
  65.   params:
  66.   - name: gitrepositoryurl
  67.     type: string
  68.   - name: branch
  69.     type: string
  70.   - name: short_sha
  71.     type: string
  72.   - name: docker_repo
  73.     type: string
  74.   - name: app_name
  75.     type: string
  76.   - name: app_ns
  77.     type: string
  78.   - name: k8s_server
  79.     type: string
  80.   # Here you can connect with CI process to realize CI/CD Pipeline
  81.   tasks:
  82.   - name: helloworld-argocd
  83.     taskRef:
  84.       name: helloworld-argocd
  85.     params:
  86.     - name: gitrepositoryurl
  87.       value: $(params.gitrepositoryurl)
  88.     - name: short_sha
  89.       value: $(params.short_sha)
  90.     - name: branch
  91.       value: $(params.branch)
  92.     - name: docker_repo
  93.       value: $(params.docker_repo)
  94.     - name: app_name
  95.       value: $(params.app_name)
  96.     - name: app_ns
  97.       value: $(params.app_ns)
  98.     - name: k8s_server
  99.       value: $(params.k8s_server)

Run Tekton Pipeline

Configure PipelineRun CRD, and kubectl apply -f in k8s cluster to run Pipeline, as follows:

  1. apiVersion: tekton.dev/v1beta1
  2. kind: PipelineRun
  3. metadata:
  4.   name: helloworld-pipeline-run-1
  5. spec:
  6.   pipelineRef:
  7.     name: helloworld-pipeline
  8.   params:
  9.   - name: gitrepositoryurl
  10.     value: https://github.com/zmberg/samples.git
  11.   - name: branch
  12.     value: hello_world
  13.   - name: short_sha
  14.     value: d92ae174b
  15.   - name: docker_repo
  16.     value: zhaomingshan/kruise
  17.   - name: app_name
  18.     value: helloworld
  19.   - name: app_ns
  20.     value: helloworld
  21.   - name: k8s_server
  22.     value: https://kubernetes.default.svc

The results can be viewed via the argo-cd cli, as follows:

guestbook

Summary

OpenKruise is more of a Kubernetes level extended capability, such as in-place upgrade, preImageDownload, etc. So many community users using OpenKruise in production environments have some additional costs, need to integrate or self-research container PaaS. The main purpose of this article is to combine some of the best Paas solutions in the community with OpenKruise, so that as many people as possible get to enjoy the dividends of cloud-native at a lower cost.

Argo-cd is a very good product for the community, and it integrate well with OpenKruise’s CRD resources, we hope that this article can intrigue more ideas from the community about “How to use OpenKruise easily”. We will try to integrate OpenKruise with other CI/CD pipelines later to better implement DevOps practice.