06-2. apiserver 高可用

本文档讲解使用 nginx 4 层透明代理功能实现 Kubernetes worker 节点组件高可用访问 kube-apiserver 集群的步骤。

注意:如果没有特殊指明,本文档的所有操作均在 zhangjun-k8s-01 节点上执行

基于 nginx 代理的 kube-apiserver 高可用方案

  • 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署且连接本机的 kube-apiserver,所以只要有一个实例正常,就可以保证高可用;
  • 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver, kube-dns 会自动解析出多个 kube-apiserver 节点的 IP,所以也是高可用的;
  • 在每个节点起一个 nginx 进程,后端对接多个 apiserver 实例,nginx 对它们做健康检查和负载均衡;
  • kubelet、kube-proxy 通过本地的 nginx(监听 127.0.0.1)访问 kube-apiserver,从而实现 kube-apiserver 的高可用;

下载和编译 nginx

下载源码:

  1. cd /opt/k8s/work
  2. wget http://nginx.org/download/nginx-1.15.3.tar.gz
  3. tar -xzvf nginx-1.15.3.tar.gz

配置编译参数:

  1. cd /opt/k8s/work/nginx-1.15.3
  2. mkdir nginx-prefix
  3. yum install -y gcc make
  4. ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  • --with-stream:开启 4 层透明转发(TCP Proxy)功能;
  • --without-xxx:关闭所有其他功能,这样生成的动态链接二进制程序依赖最小;

输出:

  1. Configuration summary
  2. + PCRE library is not used
  3. + OpenSSL library is not used
  4. + zlib library is not used
  5. nginx path prefix: "/root/tmp/nginx-1.15.3/nginx-prefix"
  6. nginx binary file: "/root/tmp/nginx-1.15.3/nginx-prefix/sbin/nginx"
  7. nginx modules path: "/root/tmp/nginx-1.15.3/nginx-prefix/modules"
  8. nginx configuration prefix: "/root/tmp/nginx-1.15.3/nginx-prefix/conf"
  9. nginx configuration file: "/root/tmp/nginx-1.15.3/nginx-prefix/conf/nginx.conf"
  10. nginx pid file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/nginx.pid"
  11. nginx error log file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/error.log"
  12. nginx http access log file: "/root/tmp/nginx-1.15.3/nginx-prefix/logs/access.log"
  13. nginx http client request body temporary files: "client_body_temp"
  14. nginx http proxy temporary files: "proxy_temp"

编译和安装:

  1. cd /opt/k8s/work/nginx-1.15.3
  2. make && make install

验证编译的 nginx

  1. cd /opt/k8s/work/nginx-1.15.3
  2. ./nginx-prefix/sbin/nginx -v

输出:

  1. nginx version: nginx/1.15.3

安装和部署 nginx

创建目录结构:

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  7. done

拷贝二进制程序:

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  7. scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
  8. ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  9. done
  • 重命名二进制文件为 kube-nginx;

配置 nginx,开启 4 层透明转发功能:

  1. cd /opt/k8s/work
  2. cat > kube-nginx.conf << \EOF
  3. worker_processes 1;
  4. events {
  5. worker_connections 1024;
  6. }
  7. stream {
  8. upstream backend {
  9. hash $remote_addr consistent;
  10. server 172.27.138.251:6443 max_fails=3 fail_timeout=30s;
  11. server 172.27.137.229:6443 max_fails=3 fail_timeout=30s;
  12. server 172.27.138.239:6443 max_fails=3 fail_timeout=30s;
  13. }
  14. server {
  15. listen 127.0.0.1:8443;
  16. proxy_connect_timeout 1s;
  17. proxy_pass backend;
  18. }
  19. }
  20. EOF
  • upstream backend 中的 server 列表为集群中各 kube-apiserver 的节点 IP,需要根据实际情况修改

分发配置文件:

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. scp kube-nginx.conf root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  7. done

配置 systemd unit 文件,启动服务

配置 kube-nginx systemd unit 文件:

  1. cd /opt/k8s/work
  2. cat > kube-nginx.service <<EOF
  3. [Unit]
  4. Description=kube-apiserver nginx proxy
  5. After=network.target
  6. After=network-online.target
  7. Wants=network-online.target
  8. [Service]
  9. Type=forking
  10. ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
  11. ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
  12. ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
  13. PrivateTmp=true
  14. Restart=always
  15. RestartSec=5
  16. StartLimitInterval=0
  17. LimitNOFILE=65536
  18. [Install]
  19. WantedBy=multi-user.target
  20. EOF

分发 systemd unit 文件:

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. scp kube-nginx.service root@${node_ip}:/etc/systemd/system/
  7. done

启动 kube-nginx 服务:

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  7. done

检查 kube-nginx 服务运行状态

  1. cd /opt/k8s/work
  2. source /opt/k8s/bin/environment.sh
  3. for node_ip in ${NODE_IPS[@]}
  4. do
  5. echo ">>> ${node_ip}"
  6. ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  7. done

确保状态为 active (running),否则查看日志,确认原因:

  1. journalctl -u kube-nginx