Cloud-Native Devops Best Practices(2) - GitOps + OpenKruise CloneSet

What is GitOps?

GitOps is an approach to continuous delivery. Its core idea is to store the declarative infrastructure and applications of an application in a Git repository.

With Git at the core of the delivery pipeline, every developer can submit Pull Requests and use Git to accelerate and simplify application deployment and maintenance tasks for Kubernetes. By using a simple tool like Git, developers can more efficiently focus on creating new features rather than operations-related tasks (e.g., application installation, configuration, migration, etc.).

argo-cd

GitOps + OpenKruise CloneSet Practice

Requirements

  • Install Kubernetes Cluster, Since v1.0.0 (alpha/beta), OpenKruise requires Kubernetes version >= 1.16.
  • Install Tekton, Reference Official Documents, Tekton is a Google open source Kubernetes native framework for creating continuous integration and continuous deployment/delivery (CI/CD) systems.
  • Install Argo-cd, Reference Official Documents, Argo-cd is a declarative GitOps continuous delivery tool for Kubernetes.

Install OpenKruise(Enable: TemplateNoDefaults)

Openkruise installed by default will inject the default value of pod / PVC template, which will conflict with the sync judgment logic of Argo CD. Therefore, when installing openkruise, you need to open gates TemplateNoDefaults, as follows:

  1. # Firstly add openkruise charts repository if you haven't do this.
  2. $ helm repo add openkruise https://openkruise.github.io/charts/
  3. # [Optional]
  4. $ helm repo update
  5. # Install the latest version.
  6. $ helm install kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"
  7. # Those that have been installed need to be upgraded
  8. $ helm upgrade kruise openkruise/kruise --set featureGates="TemplateNoDefaults=true"

CloneSet Deploy Stateless Application

CloneSet is the ability provided by OpenKruise to efficiently manage stateless applications, it is similar to the official workload: Deployment, but offers many enhancements such as InPlace Update, Batch Release, etc. Please refer to the documentation CloneSet. This article provides a helloworld http service demo, It contains Helm Charts, and the cloneSet configuration is shown below:

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: CloneSet
  3. metadata:
  4. name: helloworld-server
  5. labels:
  6. app: helloworld-server
  7. spec:
  8. updateStrategy:
  9. # CloneSet will try to in-place update Pod instead of recreating them if possible
  10. type: InPlaceIfPossible
  11. # Batch release, currently updating only one Pod
  12. partition: 1
  13. replicas: 2
  14. selector:
  15. matchLabels:
  16. app: helloworld-server
  17. template:
  18. metadata:
  19. labels:
  20. app: helloworld-server
  21. spec:
  22. containers:
  23. - name: helloworld
  24. image: "openkruise/kruise:hello_world-d92ae174b"

Argo-cd CloneSet Health Check

Configure CloneSet Argo-cd Custom CRD Health Checks, With this configuration argo-cd is able to perform a healthy check of the CloneSet, such as whether the CloneSet is published and whether the Pods are ready, as follows:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. labels:
  5. app.kubernetes.io/name: argocd-cm
  6. app.kubernetes.io/part-of: argocd
  7. name: argocd-cm
  8. namespace: argocd
  9. data:
  10. resource.customizations.health.apps.kruise.io_CloneSet: |
  11. hs = {}
  12. -- if paused
  13. if obj.spec.updateStrategy.paused then
  14. hs.status = "Suspended"
  15. hs.message = "CloneSet is Suspended"
  16. return hs
  17. end
  18. -- check cloneSet status
  19. if obj.status ~= nil then
  20. if obj.status.observedGeneration < obj.metadata.generation then
  21. hs.status = "Progressing"
  22. hs.message = "Waiting for rollout to finish: observed cloneSet generation less then desired generation"
  23. return hs
  24. end
  25. if obj.status.updatedReplicas < obj.spec.replicas then
  26. hs.status = "Progressing"
  27. hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  28. return hs
  29. end
  30. if obj.status.updatedReadyReplicas < obj.status.updatedReplicas then
  31. hs.status = "Progressing"
  32. hs.message = "Waiting for rollout to finish: replicas hasn't finished updating..."
  33. return hs
  34. end
  35. hs.status = "Healthy"
  36. return hs
  37. end
  38. -- if status == nil
  39. hs.status = "Progressing"
  40. hs.message = "Waiting for cloneSet"
  41. return hs

OpenKruise internal CRD resources in addition to CloneSet, others such as: Advanced StatefulSet, SidecarSet, etc. can be similar to the above way to achieve Custom Resource Health.

Tekton Pipeline + Argo-cd

Argo-CD together with Tekton Pipeline is a popular DevOps practice and integrate well with CI process. Such practice requires storing the Argo-cd admin secret in K8S Secret CRD (method of obtaining secret), which in turn can be used in Tekton Pipeline, as follows:

  1. apiVersion: v1
  2. data:
  3. # argo-cd admin secret
  4. username: xxxxx
  5. password: xxxxx
  6. server: xxxxx
  7. kind: Secret
  8. metadata:
  9. name: argosecret
  10. ---
  11. apiVersion: tekton.dev/v1beta1
  12. kind: Task
  13. metadata:
  14. labels:
  15. app: helloworld
  16. name: helloworld-argocd
  17. spec:
  18. params:
  19. - name: gitrepositoryurl
  20. type: string
  21. - name: branch
  22. type: string
  23. - name: short_sha
  24. type: string
  25. - name: docker_repo
  26. type: string
  27. - name: app_name
  28. type: string
  29. - name: app_ns
  30. type: string
  31. - name: k8s_server
  32. type: string
  33. steps:
  34. - name: argocd-deploy
  35. image: argoproj/argocd:latest
  36. command:
  37. - sh
  38. args:
  39. - '-ce'
  40. - >
  41. set -e
  42. echo "upgrade app $(params.app_name)"; username=`cat /var/secret/username`; password=`cat /var/secret/password`; server=`cat /var/secret/server`;
  43. argocd login ${server} --insecure --username ${username} --password ${password}
  44. argocd app create $(params.app_name) --upsert --repo $(params.gitrepositoryurl) --path $(params.app_name)/charts --dest-namespace $(params.app_ns) --dest-server $(params.k8s_server) --revision $(params.branch) --helm-set image.repository=$(params.docker_repo) --helm-set image.tag=$(params.branch)-$(params.short_sha) --helm-set installation.namespace=$(params.app_ns)
  45. argocd app list; argocd app sync $(params.app_name)
  46. argocd app wait $(params.app_name) --health
  47. volumeMounts:
  48. - name: argocd-secret
  49. mountPath: "/var/secret"
  50. volumes:
  51. - name: argocd-secret
  52. secret:
  53. secretName: argosecret
  54. ---
  55. apiVersion: tekton.dev/v1beta1
  56. kind: Pipeline
  57. metadata:
  58. name: helloworld-pipeline
  59. spec:
  60. params:
  61. - name: gitrepositoryurl
  62. type: string
  63. - name: branch
  64. type: string
  65. - name: short_sha
  66. type: string
  67. - name: docker_repo
  68. type: string
  69. - name: app_name
  70. type: string
  71. - name: app_ns
  72. type: string
  73. - name: k8s_server
  74. type: string
  75. # Here you can connect with CI process to realize CI/CD Pipeline
  76. tasks:
  77. - name: helloworld-argocd
  78. taskRef:
  79. name: helloworld-argocd
  80. params:
  81. - name: gitrepositoryurl
  82. value: $(params.gitrepositoryurl)
  83. - name: short_sha
  84. value: $(params.short_sha)
  85. - name: branch
  86. value: $(params.branch)
  87. - name: docker_repo
  88. value: $(params.docker_repo)
  89. - name: app_name
  90. value: $(params.app_name)
  91. - name: app_ns
  92. value: $(params.app_ns)
  93. - name: k8s_server
  94. value: $(params.k8s_server)

Run Tekton Pipeline

Configure PipelineRun CRD, and kubectl apply -f in k8s cluster to run Pipeline, as follows:

  1. apiVersion: tekton.dev/v1beta1
  2. kind: PipelineRun
  3. metadata:
  4. name: helloworld-pipeline-run-1
  5. spec:
  6. pipelineRef:
  7. name: helloworld-pipeline
  8. params:
  9. - name: gitrepositoryurl
  10. value: https://github.com/zmberg/samples.git
  11. - name: branch
  12. value: hello_world
  13. - name: short_sha
  14. value: d92ae174b
  15. - name: docker_repo
  16. value: zhaomingshan/kruise
  17. - name: app_name
  18. value: helloworld
  19. - name: app_ns
  20. value: helloworld
  21. - name: k8s_server
  22. value: https://kubernetes.default.svc

The results can be viewed via the argo-cd cli, as follows:

guestbook

Summary

OpenKruise is more of a Kubernetes level extended capability, such as in-place upgrade, preImageDownload, etc. So many community users using OpenKruise in production environments have some additional costs, need to integrate or self-research container PaaS. The main purpose of this article is to combine some of the best Paas solutions in the community with OpenKruise, so that as many people as possible get to enjoy the dividends of cloud-native at a lower cost.

Argo-cd is a very good product for the community, and it integrate well with OpenKruise’s CRD resources, we hope that this article can intrigue more ideas from the community about “How to use OpenKruise easily”. We will try to integrate OpenKruise with other CI/CD pipelines later to better implement DevOps practice.