Jenkins优化Kubernetes部署流水线

在上一节，我们实现了全链路的部署流水线。

本节，我们将继续完善、优化部署水线。

Gradle加速

首先，在之前的定制Agent中，我们使用了Gradle(Maven)的默认仓库。

由于众所周知的原因，默认仓库的速度很慢、不稳定。

这回严重降低打包流水线的速度，我们对这一问题进行优化。

修改Agent的Dockerfile如下，增加Gradle仓库配置：

  1. FROM jenkins/inbound-agent:latest-jdk8
  3. ENV GRADLE_VERSION=7.2
  4. ENV K8S_VERSION=v1.22.3
  5. ENV DOCKER_CHANNEL stable
  6. ENV DOCKER_VERSION 18.06.3-ce 
  8. # tool
  9. USER root
  10. RUN apt-get update && \
  11.     apt-get install -y curl unzip sudo && \
  12.     apt-get clean
  14. # docker
  15. RUN curl -fsSL "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz" \
  16.   | tar -xzC /usr/local/bin --strip=1 docker/docker
  18. # gradle
  19. RUN curl -skL -o /tmp/gradle-bin.zip https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip && \
  20.     mkdir -p /opt/gradle && \
  21.     unzip -q /tmp/gradle-bin.zip -d /opt/gradle && \
  22.     ln -sf /opt/gradle/gradle-$GRADLE_VERSION/bin/gradle /usr/local/bin/gradle
  24. RUN chown -R 1001:0 /opt/gradle && \
  25.     chmod -R g+rw /opt/gradle
  27. # kubectl
  28. RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$K8S_VERSION/bin/linux/amd64/kubectl
  29. RUN chmod +x ./kubectl
  30. RUN mv ./kubectl /usr/local/bin
  33. # add jenkins user to sudoer without password 
  34. RUN usermod -aG sudo jenkins 
  35. RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
  37. # jenkins
  38. USER jenkins
  40. # gradle mirror
  41. ENV GRADLE_CONFIG_DIR=/home/jenkins/.gradle 
  42. RUN mkdir ${GRADLE_CONFIG_DIR}
  43. RUN echo "Ly8gcHJvamVjdAphbGxwcm9qZWN0c3sKICAgIHJlcG9zaXRvcmllcyB7CgltYXZlbkxvY2FsKCkKICAgICAgICBtYXZlbiB7IHVybCAnaHR0cHM6Ly9tYXZlbi5hbGl5dW4uY29tL3JlcG9zaXRvcnkvcHVibGljLycgfQogICAgICAgIG1hdmVuIHsgdXJsICdodHRwczovL21hdmVuLmFsaXl1bi5jb20vcmVwb3NpdG9yeS9qY2VudGVyLycgfQogICAgICAgIG1hdmVuIHsgdXJsICdodHRwczovL21hdmVuLmFsaXl1bi5jb20vcmVwb3NpdG9yeS9nb29nbGUvJyB9CiAgICAgICAgbWF2ZW4geyB1cmwgJ2h0dHBzOi8vbWF2ZW4uYWxpeXVuLmNvbS9yZXBvc2l0b3J5L2dyYWRsZS1wbHVnaW4vJyB9CiAgICAgICAgbWF2ZW4geyB1cmwgJ2h0dHBzOi8vaml0cGFjay5pby8nIH0KICAgIH0KfQoKLy8gcGx1Z2luCnNldHRpbmdzRXZhbHVhdGVkIHsgc2V0dGluZ3MgLT4KICAgIHNldHRpbmdzLnBsdWdpbk1hbmFnZW1lbnQgewogICAgICAgIC8vIFByaW50IHJlcG9zaXRvcmllcyBjb2xsZWN0aW9uCiAgICAgICAgLy8gcHJpbnRsbiAiUmVwb3NpdG9yaWVzIG5hbWVzOiAiICsgcmVwb3NpdG9yaWVzLmdldE5hbWVzKCkKCiAgICAgICAgLy8gQ2xlYXIgcmVwb3NpdG9yaWVzIGNvbGxlY3Rpb24KICAgICAgICByZXBvc2l0b3JpZXMuY2xlYXIoKQoKICAgICAgICAvLyBBZGQgbXkgQXJ0aWZhY3RvcnkgbWlycm9yCiAgICAgICAgcmVwb3NpdG9yaWVzIHsKCSAgICBtYXZlbkxvY2FsKCkKICAgICAgICAgICAgbWF2ZW4gewogICAgICAgICAgICAgICAgdXJsICJodHRwczovL21hdmVuLmFsaXl1bi5jb20vcmVwb3NpdG9yeS9ncmFkbGUtcGx1Z2luLyIKICAgICAgICAgICAgfQogICAgICAgIH0KICAgIH0KfQo=" | base64 --decode > ${GRADLE_CONFIG_DIR}/init.gradle

如上所示，在打包的最后环节：

  • 添加.gradle目录

  • 创建init.gradle脚本

  • 由于Dockerfile的语法格式限制，我们将配置文件编码为Base64再写入

配置文件的原文如下：

  1. // project
  2. allprojects{
  3.     repositories {
  4.     mavenLocal()
  5.         maven { url 'https://maven.aliyun.com/repository/public/' }
  6.         maven { url 'https://maven.aliyun.com/repository/jcenter/' }
  7.         maven { url 'https://maven.aliyun.com/repository/google/' }
  8.         maven { url 'https://maven.aliyun.com/repository/gradle-plugin/' }
  9.         maven { url 'https://jitpack.io/' }
  10.     }
  11. }
  13. // plugin
  14. settingsEvaluated { settings ->
  15.     settings.pluginManagement {
  16.         // Print repositories collection
  17.         // println "Repositories names: " + repositories.getNames()
  19.         // Clear repositories collection
  20.         repositories.clear()
  22.         // Add my Artifactory mirror
  23.         repositories {
  24.         mavenLocal()
  25.             maven {
  26.                 url "https://maven.aliyun.com/repository/gradle-plugin/"
  27.             }
  28.         }
  29.     }
  30. }

我们使用新镜像重启Agent，会发现编译环节由1分钟缩短到10秒内。

  • 滚动升级

在之前构建的版本中，我们只考虑了部署，没有考虑升级情况。

可以修改JenkinsFile，采用”yaml + kubectl apply”的方式，让部署支持滚动升级。

  1. pipeline {
  2.     agent any
  4.     environment {
  5.         project = "coder4/homs-start"
  6.     }
  8.     stages {
  9.         stage('git') {
  10.             steps {
  11.                 git credentialsId: 'GITEE', url: 'git@gitee.com:/'+ project + '.git', branch: 'master'
  12.             }
  13.         }
  15.         stage('gradle') {
  16.             steps {
  17.                 sh "gradle build"
  18.             }    
  19.         }
  21.         stage('docker image build') {
  22.             steps {
  23.                 sh '''
  24.                 # get right jar
  25.                 jarPath=$(du -a ./build/libs/* | sort -n -r | head -n 1 | cut -f2-)
  26.                 jarFile=$( echo ${jarPath##*/} )
  28.                 # make Dockerfile
  29. cat <<EOF > Dockerfile
  30. FROM openjdk:8
  31. COPY $jarPath $jarFile
  32. ENTRYPOINT ["java","-jar","/$jarFile"]
  33. EOF
  34.                 # build Docker image
  35.                 sudo docker build -t coder4/${JOB_NAME}:${BUILD_NUMBER} .
  37.                 # push to docker hub
  38.                 sudo docker push coder4/${JOB_NAME}:${BUILD_NUMBER}
  39.                 '''
  40.             }
  41.         }
  43.         stage('k8s') { 
  44.             steps {
  45.                 withKubeConfig([credentialsId: "60a8e9d2-0212-4ff4-aa98-f46fced97121",serverUrl: "https://kubernetes:6443"]) {
  46.                     sh """
  47.                     # prepare deployment yaml
  48. cat <<EOF  | kubectl apply -f -
  49. apiVersion: apps/v1
  50. kind: Deployment
  51. metadata:
  52.   name: ${JOB_NAME}-deployment
  53.   labels:
  54.     app: ${JOB_NAME}
  55. spec:
  56.   selector:
  57.     matchLabels:
  58.       app: ${JOB_NAME}
  59.   replicas: 1
  60.   strategy:
  61.     type: RollingUpdate
  62.   template:
  63.     metadata:
  64.       labels:
  65.         app: ${JOB_NAME}
  66.     spec:
  67.       containers:
  68.         - name: ${JOB_NAME}-server
  69.           image: coder4/${JOB_NAME}:${BUILD_NUMBER}
  70.           ports:
  71.             - containerPort: 8080
  72.                     """
  73.                 }
  74.             }
  75.         }
  76.     }
  77. }

经过上述改造后，我们可以随时滚动升级新版本了。

支持回滚操作

在新版本发布后，可能会遇到故障，需要回滚的情况，这也需要流水线支持这一功能。

我们采用”Parameterized Project”的方式，来设定参数。

首先，修改当前项目的配置，勾选”This project is parameterized”。

接着，安装插件“Active Choice”，以便开启Groovy脚本的“动态参数”。

加下来，我们添加3个参数

  1. Active Choices Parameter，参数名”JobName”

代码、截图如下：

  1. m = Thread.currentThread().toString() =~ /job\/(.*)\/build/
  2. return [m[0][1]]

f

  1. Choose Parameter，参数名”Action”，固定两个选项：Deploy、Rollback

代码和截图如下：

f

  1. Active Choices Reactive Parameter，参数名”RollbackVersion”

需要配置Referenced parameters为”Action,JobName”

代码和截图如下：

  1. if (Action.equals('Deploy')) {
  2.     return []
  3. } else {
  4.     return jenkins.model.Jenkins.instance.getJob(JobName).builds.findAll{ it.result == hudson.model.Result.SUCCESS }.collect{ "$it.number".toString() }
  5. }

f

经过上述设置，我们的项目拥有了3个可输入参数，如下图所示：

f

其中：

JobName：项目名

Action：决定了是部署 or 回滚

RollbackVersion：仅当回滚时生效，决定了要回滚到哪个版本

除此之外，我们还需要对JenkinsFile进行改造，如下：

  1. pipeline {
  2.     agent any
  4.     stages {
  5.         stage('git') {
  6.             steps {
  7.                 script {
  8.                     if (params.Action.equals("Rollback")) {
  9.                         echo "Skip in Rollback"
  10.                     } else {
  11.                         git credentialsId: 'GITEE', url: 'git@gitee.com:/coder4/'+ env.JOB_NAME + '.git', branch: 'master'
  12.                     }
  13.                 }
  14.             }
  15.         }
  17.         stage('gradle') {
  18.             steps {
  19.                 script {
  20.                     if (params.Action.equals("Rollback")) {
  21.                         echo "Skip in Rollback"
  22.                     } else {
  23.                         sh "gradle build"
  24.                     }
  25.                 }
  26.             }    
  27.         }
  29.         stage('docker image build') {
  30.             steps {
  31.                 script {
  32.                     if (params.Action.equals("Rollback")) {
  33.                         echo "Skip in Rollback"
  34.                     } else {
  35.                         sh '''
  36.                 # get right jar
  37.                 jarPath=$(du -a ./build/libs/* | sort -n -r | head -n 1 | cut -f2-)
  38.                 jarFile=$( echo ${jarPath##*/} )
  40.                 # make Dockerfile
  41. cat <<EOF > Dockerfile
  42. FROM openjdk:8
  43. COPY $jarPath $jarFile
  44. ENTRYPOINT ["java","-jar","/$jarFile"]
  45. EOF
  46.                 # build Docker image
  47.                 sudo docker build -t coder4/${JOB_NAME}:${BUILD_NUMBER} .
  49.                 # push to docker hub
  50.                 sudo docker push coder4/${JOB_NAME}:${BUILD_NUMBER}
  51.                 '''
  52.                     }
  53.                 }
  54.             }
  55.         }
  57.         stage('k8s') { 
  58.             steps {
  59.                 script {
  60.                     env.DEPLOY_VERSION = params.Action.equals("Rollback") ? params.RollbackVersion : env.BUILD_NUMBER
  63.                     withKubeConfig([credentialsId: "60a8e9d2-0212-4ff4-aa98-f46fced97121",serverUrl: "https://kubernetes:6443"]) {
  64.                         sh """
  66. echo "Kubernetes Deploy $JOB_NAME Version $DEPLOY_VERSION"
  68. # prepare deployment yaml
  69. cat <<EOF  | kubectl apply -f -
  70. apiVersion: apps/v1
  71. kind: Deployment
  72. metadata:
  73.   name: ${JOB_NAME}-deployment
  74.   labels:
  75.     app: ${JOB_NAME}
  76. spec:
  77.   selector:
  78.     matchLabels:
  79.       app: ${JOB_NAME}
  80.   replicas: 1
  81.   strategy:
  82.     type: RollingUpdate
  83.   template:
  84.     metadata:
  85.       labels:
  86.         app: ${JOB_NAME}
  87.     spec:
  88.       containers:
  89.         - name: ${JOB_NAME}-server
  90.           image: coder4/${JOB_NAME}:${DEPLOY_VERSION}
  91.           ports:
  92.             - containerPort: 8080
  93.                     """
  94.                     }
  95.                 }
  96.             }
  97.         }
  98.     }
  99. }

我们来试验一下成果，首先，执行新部署：执行成功，版本号111，耗时21s

  1. kubectl describe pod homs-start-deployment-644677f984-bksl9
  2. Name:         homs-start-deployment-644677f984-bksl9
  3. Namespace:    default
  4. Priority:     0
  5. Node:         minikube/192.168.49.2
  6. Start Time:   Thu, 11 Nov 2021 19:06:25 +0800
  7. Labels:       app=homs-start
  8.               pod-template-hash=644677f984
  9. Annotations:  <none>
  10. Status:       Running
  11. IP:           172.17.0.4
  12. IPs:
  13.   IP:           172.17.0.4
  14. Controlled By:  ReplicaSet/homs-start-deployment-644677f984
  15. Containers:
  16.   homs-start-server:
  17.     Container ID:   docker://279e11005931dfd8aa876134bb2441294a768766261aeb0bb88b5004047f5060
  18.     Image:          coder4/homs-start:111
  19.     Image ID:       docker-pullable://coder4/homs-start@sha256:526640caca84a10254e42ad12dd617eaf45c75c17b4ebb7731fe623509938e5c
  20.     Port:           8080/TCP
  21.     Host Port:      0/TCP
  22.     State:          Running
  23.       Started:      Thu, 11 Nov 2021 19:06:31 +0800
  24.     Ready:          True
  25.     Restart Count:  0
  26.     Environment:    <none>
  27.     Mounts:
  28.       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gkpv7 (ro)
  29. Conditions:
  30.   Type              Status
  31.   Initialized       True 
  32.   Ready             True 
  33.   ContainersReady   True 
  34.   PodScheduled      True 
  35. Volumes:
  36.   kube-api-access-gkpv7:
  37.     Type:                    Projected (a volume that contains injected data from multiple sources)
  38.     TokenExpirationSeconds:  3607
  39.     ConfigMapName:           kube-root-ca.crt
  40.     ConfigMapOptional:       <nil>
  41.     DownwardAPI:             true
  42. QoS Class:                   BestEffort
  43. Node-Selectors:              <none>
  44. Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  45.                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  46. Events:
  47.   Type    Reason     Age   From               Message
  48.   ----    ------     ----  ----               -------
  49.   Normal  Scheduled  37s   default-scheduler  Successfully assigned default/homs-start-deployment-644677f984-bksl9 to minikube
  50.   Normal  Pulling    37s   kubelet            Pulling image "coder4/homs-start:111"
  51.   Normal  Pulled     31s   kubelet            Successfully pulled image "coder4/homs-start:111" in 5.781019732s
  52.   Normal  Created    31s   kubelet            Created container homs-start-server
  53.   Normal  Started    31s   kubelet            Started container homs-start-server

接下来，我们回滚到107版本，由于机器上有镜像，因此只耗时1s。

  1. kubectl describe pod homs-start-deployment-5bf947768c-dt8w2
  2. Name:         homs-start-deployment-5bf947768c-dt8w2
  3. Namespace:    default
  4. Priority:     0
  5. Node:         minikube/192.168.49.2
  6. Start Time:   Thu, 11 Nov 2021 18:49:22 +0800
  7. Labels:       app=homs-start
  8.               pod-template-hash=5bf947768c
  9. Annotations:  <none>
  10. Status:       Running
  11. IP:           172.17.0.5
  12. IPs:
  13.   IP:           172.17.0.5
  14. Controlled By:  ReplicaSet/homs-start-deployment-5bf947768c
  15. Containers:
  16.   homs-start-server:
  17.     Container ID:   docker://bc626494af343b6d56b707258e03a85ae668abb21dcc3ca2b72d6239e3b56b3d
  18.     Image:          coder4/homs-start:107
  19.     Image ID:       docker-pullable://coder4/homs-start@sha256:526640caca84a10254e42ad12dd617eaf45c75c17b4ebb7731fe623509938e5c
  20.     Port:           8080/TCP
  21.     Host Port:      0/TCP
  22.     State:          Running
  23.       Started:      Thu, 11 Nov 2021 18:49:27 +0800
  24.     Ready:          True
  25.     Restart Count:  0
  26.     Environment:    <none>
  27.     Mounts:
  28.       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dt7g2 (ro)
  29. Conditions:
  30.   Type              Status
  31.   Initialized       True 
  32.   Ready             True 
  33.   ContainersReady   True 
  34.   PodScheduled      True 
  35. Volumes:
  36.   kube-api-access-dt7g2:
  37.     Type:                    Projected (a volume that contains injected data from multiple sources)
  38.     TokenExpirationSeconds:  3607
  39.     ConfigMapName:           kube-root-ca.crt
  40.     ConfigMapOptional:       <nil>
  41.     DownwardAPI:             true
  42. QoS Class:                   BestEffort
  43. Node-Selectors:              <none>
  44. Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  45.                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  46. Events:
  47.   Type    Reason     Age   From               Message
  48.   ----    ------     ----  ----               -------
  49.   Normal  Scheduled  16m   default-scheduler  Successfully assigned default/homs-start-deployment-5bf947768c-dt8w2 to minikube
  50.   Normal  Pulling    16m   kubelet            Pulling image "coder4/homs-start:107"
  51.   Normal  Pulled     16m   kubelet            Successfully pulled image "coder4/homs-start:107" in 3.365201023s
  52.   Normal  Created    16m   kubelet            Created container homs-start-server
  53.   Normal  Started    16m   kubelet            Started container homs-start-server

在本文中，我们围绕编译、镜像进行了优化，但这还远没有达到”完美”的程度。

我提一些思路，供大家参考：

  1. docker镜像瘦身：打Dokcer镜像时，其实无需将jdk+ jar包一起打，可以只打jar包。在生成Deployment时，通过Pod的init container模式，将jar包拷贝进jdk的运行容器中，从而完成启动。

  2. 回滚版本选择优化：在前面的实现中，我们筛选了所有成功部署过的版本，将其做为可回滚的版本，但这其中的一部分，实际是通过”回滚”的方式部署成功的，在镜像仓库中，并没有与之对应的镜像版本。我们可以拉取镜像仓库中可用的版本，来实现回滚。

  3. 镜像版本优化：目前采用的是Job的”Build Version”做为镜像版本，可以再此基础上，追加Git版本号，以便区分代码拉取。

  4. 支持多分之：当前，我们默认用的是master分之，应当可以通过参数的方式，支持不同分之的修改。

  5. JenkinsFile共享：目前的JenkinsFile是直接配置在项目中的，如果微服务项目很多，逐一配置势必很麻烦，可以通过 “Jenkins Shared Library”的方式，在多项目间共享脚本配置。