kubeadm

Kubernetes 一键部署脚本（使用 docker 运行时）

  1. # on master
  2. export USE_MIRROR=true #国内用户必须使用MIRROR
  3. git clone https://github.com/feiskyer/ops
  4. cd ops
  5. kubernetes/install-kubernetes.sh
  6. # 记住控制台输出的 TOEKN 和 MASTER 地址，在其他 Node 安装时会用到
  8. # on all nodes
  9. git clone https://github.com/feiskyer/ops
  10. cd ops
  11. # Setup token and CIDR first.
  12. # replace this with yours.
  13. export TOKEN="xxxx"
  14. export MASTER_IP="x.x.x.x"
  15. export CONTAINER_CIDR="10.244.2.0/24"
  17. # Setup and join the new node.
  18. ./kubernetes/add-node.sh

以下是详细的 kubeadm 部署集群步骤。

初始化系统

所有机器都需要初始化 docker 和 kubelet。

ubuntu

  1. # for ubuntu 16.04
  2. apt-get update
  3. apt-get install -y apt-transport-https ca-certificates curl software-properties-common
  4. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
  5. add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
  6. apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
  8. apt-get update && apt-get install -y apt-transport-https curl
  9. curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
  10.     cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
  11. deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
  12. EOF
  13. apt-get update
  14. apt-get install -y kubelet kubeadm kubectl

CentOS

  1. yum install -y docker
  2. systemctl enable docker && systemctl start docker
  4. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  5. [kubernetes]
  6. name=Kubernetes
  7. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  8. enabled=1
  9. gpgcheck=1
  10. repo_gpgcheck=1
  11. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  12. EOF
  13. setenforce 0
  14. sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
  15. yum install -y kubelet kubeadm kubectl
  16. systemctl enable kubelet && systemctl start kubelet

安装 master

  1. # --api-advertise-addresses <ip-address>
  2. # for flannel, setup --pod-network-cidr 10.244.0.0/16
  3. kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version latest
  5. # enable schedule pods on the master
  6. export KUBECONFIG=/etc/kubernetes/admin.conf
  7. kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-

如果需要修改 kubernetes 服务的配置选项，则需要创建一个 kubeadm 配置文件，其格式为

  1. apiVersion: kubeadm.k8s.io/v1alpha3
  2. kind: InitConfiguration
  3. bootstrapTokens:
  4. - token: "9a08jv.c0izixklcxtmnze7"
  5.   description: "kubeadm bootstrap token"
  6.   ttl: "24h"
  7. - token: "783bde.3f89s0fje9f38fhf"
  8.   description: "another bootstrap token"
  9.   usages:
  10.   - signing
  11.   groups:
  12.   - system:anonymous
  13. nodeRegistration:
  14.   name: "ec2-10-100-0-1"
  15.   criSocket: "/var/run/dockershim.sock"
  16.   taints:
  17.   - key: "kubeadmNode"
  18.     value: "master"
  19.     effect: "NoSchedule"
  20.   kubeletExtraArgs:
  21.     cgroupDriver: "cgroupfs"
  22. apiEndpoint:
  23.   advertiseAddress: "10.100.0.1"
  24.   bindPort: 6443
  25. ---
  26. apiVersion: kubeadm.k8s.io/v1alpha3
  27. kind: ClusterConfiguration
  28. etcd:
  29.   # one of local or external
  30.   local:
  31.     image: "k8s.gcr.io/etcd-amd64:3.2.18"
  32.     dataDir: "/var/lib/etcd"
  33.     extraArgs:
  34.       listen-client-urls: "http://10.100.0.1:2379"
  35.     serverCertSANs:
  36.     -  "ec2-10-100-0-1.compute-1.amazonaws.com"
  37.     peerCertSANs:
  38.     - "10.100.0.1"
  39.   external:
  40.     endpoints:
  41.     - "10.100.0.1:2379"
  42.     - "10.100.0.2:2379"
  43.     caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
  44.     certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
  45.     certKey: "/etcd/kubernetes/pki/etcd/etcd.key"
  46. networking:
  47.   serviceSubnet: "10.96.0.0/12"
  48.   podSubnet: "10.100.0.1/24"
  49.   dnsDomain: "cluster.local"
  50. kubernetesVersion: "v1.12.0"
  51. controlPlaneEndpoint: "10.100.0.1:6443"
  52. apiServerExtraArgs:
  53.   authorization-mode: "Node,RBAC"
  54. controlManagerExtraArgs:
  55.   node-cidr-mask-size: 20
  56. schedulerExtraArgs:
  57.   address: "10.100.0.1"
  58. apiServerExtraVolumes:
  59. - name: "some-volume"
  60.   hostPath: "/etc/some-path"
  61.   mountPath: "/etc/some-pod-path"
  62.   writable: true
  63.   pathType: File
  64. controllerManagerExtraVolumes:
  65. - name: "some-volume"
  66.   hostPath: "/etc/some-path"
  67.   mountPath: "/etc/some-pod-path"
  68.   writable: true
  69.   pathType: File
  70. schedulerExtraVolumes:
  71. - name: "some-volume"
  72.   hostPath: "/etc/some-path"
  73.   mountPath: "/etc/some-pod-path"
  74.   writable: true
  75.   pathType: File
  76. apiServerCertSANs:
  77. - "10.100.1.1"
  78. - "ec2-10-100-0-1.compute-1.amazonaws.com"
  79. certificatesDir: "/etc/kubernetes/pki"
  80. imageRepository: "k8s.gcr.io"
  81. unifiedControlPlaneImage: "k8s.gcr.io/controlplane:v1.12.0"
  82. auditPolicy:
  83.   # https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy
  84.   path: "/var/log/audit/audit.json"
  85.   logDir: "/var/log/audit"
  86.   logMaxAge: 7 # in days
  87. featureGates:
  88.   selfhosting: false
  89. clusterName: "example-cluster"

注意：JoinConfiguration 重命名自 v1alpha2 API 中的 NodeConfiguration，而 InitConfiguration 重命名自 v1alpha2 API 中的 MasterConfiguration。

然后，在初始化 master 的时候指定 kubeadm.yaml 的路径：

  1. kubeadm init --config ./kubeadm.yaml

配置 Network plugin

CNI bridge

  1. mkdir -p /etc/cni/net.d
  2. cat >/etc/cni/net.d/10-mynet.conf <<-EOF
  3. {
  4.     "cniVersion": "0.3.0",
  5.     "name": "mynet",
  6.     "type": "bridge",
  7.     "bridge": "cni0",
  8.     "isGateway": true,
  9.     "ipMasq": true,
  10.     "ipam": {
  11.         "type": "host-local",
  12.         "subnet": "10.244.0.0/16",
  13.         "routes": [
  14.             {"dst": "0.0.0.0/0"}
  15.         ]
  16.     }
  17. }
  18. EOF
  19. cat >/etc/cni/net.d/99-loopback.conf <<-EOF
  20. {
  21.     "cniVersion": "0.3.0",
  22.     "type": "loopback"
  23. }
  24. EOF

flannel

注意：需要 kubeadm init 时设置 --pod-network-cidr=10.244.0.0/16

  1. kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

weave

  1. sysctl net.bridge.bridge-nf-call-iptables=1
  2. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

calico

注意：需要 kubeadm init 时设置 --pod-network-cidr=192.168.0.0/16

  1. kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
  2. kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

添加 Node

  1. kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

跟 Master 一样，添加 Node 的时候也可以自定义 Kubernetes 服务的选项，格式为

  1. apiVersion: kubeadm.k8s.io/v1alpha2
  2. caCertPath: /etc/kubernetes/pki/ca.crt
  3. clusterName: kubernetes
  4. discoveryFile: ""
  5. discoveryTimeout: 5m0s
  6. discoveryToken: abcdef.0123456789abcdef
  7. discoveryTokenAPIServers:
  8. - kube-apiserver:6443
  9. discoveryTokenUnsafeSkipCAVerification: true
  10. kind: NodeConfiguration
  11. nodeRegistration:
  12.   criSocket: /var/run/dockershim.sock
  13.   name: thegopher
  14. tlsBootstrapToken: abcdef.0123456789abcdef
  15. token: abcdef.0123456789abcdef

在把 Node 加入集群的时候，指定 NodeConfiguration 配置文件的路径

  1. kubeadm join --config ./nodeconfig.yml --token $token ${master_ip}

Cloud Provider

默认情况下，kubeadm 不包括 Cloud Provider 的配置，在 Azure 或者 AWS 等云平台上运行时，还需要配置 Cloud Provider。如

  1. kind: MasterConfiguration
  2. apiVersion: kubeadm.k8s.io/v1alpha2
  3. apiServerExtraArgs:
  4.   cloud-provider: "{cloud}"
  5.   cloud-config: "{cloud-config-path}"
  6. apiServerExtraVolumes:
  7. - name: cloud
  8.   hostPath: "{cloud-config-path}"
  9.   mountPath: "{cloud-config-path}"
  10. controllerManagerExtraArgs:
  11.   cloud-provider: "{cloud}"
  12.   cloud-config: "{cloud-config-path}"
  13. controllerManagerExtraVolumes:
  14. - name: cloud
  15.   hostPath: "{cloud-config-path}"
  16.   mountPath: "{cloud-config-path}"

删除安装

  1. # drain and delete the node first
  2. kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  3. kubectl delete node <node name>
  5. # then reset kubeadm
  6. kubeadm reset

动态升级

kubeadm v1.8 开始支持动态升级，升级步骤为

  • 首先上传 kubeadm 配置，如 kubeadm config upload from-flags [flags]（使用命令行参数）或 kubeadm config upload from-file --config [config]（使用配置文件）
  • 在 master 上检查新版本 kubeadm upgrade plan， 当有新版本（如 v1.8.0）时，执行 kubeadm upgrade apply v1.8.0 升级控制平面
  • 手动 升级 CNI 插件（如果有新版本的话）
  • 添加自动证书回滚的 RBAC 策略 kubectl create clusterrolebinding kubeadm:node-autoapprove-certificate-rotation --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
  • 最后升级 kubelet
  1. $ kubectl drain $HOST --ignore-daemonsets
  3. # 升级软件包
  4. $ apt-get update
  5. $ apt-get upgrade
  6. # CentOS 上面执行 yum 升级
  7. # $ yum update
  9. $ kubectl uncordon $HOST

手动升级

kubeadm v1.7 以及以前的版本不支持动态升级，但可以手动升级。

升级 Master

假设你已经有一个使用 kubeadm 部署的 Kubernetes v1.6 集群，那么升级到 v1.7 的方法为：

  1. 升级安装包 apt-get upgrade && apt-get update
  2. 重启 kubelet systemctl restart kubelet
  3. 删除 kube-proxy DaemonSet KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete daemonset kube-proxy -n kube-system
  4. kubeadm init —skip-preflight-checks —kubernetes-version v1.7.1
  5. 更新 CNI 插件

升级 Node

  1. 升级安装包 apt-get upgrade && apt-get update
  2. 重启 kubelet systemctl restart kubelet

安全选项

默认情况下，kubeadm 会开启 Node 客户端证书的自动批准，如果不需要的话可以选择关闭，关闭方法为

  1. $ kubectl delete clusterrole kubeadm:node-autoapprove-bootstrap

关闭后，增加新的 Node 时，kubeadm join 会阻塞等待管理员手动批准，匹配方法为

  1. $ kubectl get csr
  2. NAME                                                   AGE       REQUESTOR                 CONDITION
  3. node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   18s       system:bootstrap:878f07   Pending
  5. $ kubectl certificate approve node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ
  6. certificatesigningrequest "node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ" approved
  8. $ kubectl get csr
  9. NAME                                                   AGE       REQUESTOR                 CONDITION
  10. node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   1m        system:bootstrap:878f07   Approved,Issued

参考文档