Kubernetes 完全教程

Kubernetes 安装和运维

王渊命 @jolestar



  1. Kubernetes 的安装
  2. Kubernetes 的组件和配置介绍
  3. Kubernetes 的高可用
  4. Kubernetes 的使用以及运维

Kubernetes 的安装 — 条条大路通罗马

  1. 云服务商托管 GCE(Google),AWS,Azure,Bluemix,QingCloud 等。
  2. 自定义安装 kops, ansible, salt, juju
  3. kubeadm

准备基础的 VM 镜像

  1. Ubuntu 16.04.3
  2. 安装 docker 以及基础工具包
  1. apt-get update
  2. apt-get install -y ebtables socat apt-transport-https bash-completion ntp wget
  3. apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
  4. apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'
  5. apt-get update
  6. apt-cache policy docker-engine
  7. apt-get install -y docker-engine
  8. DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

安装 kubelet, kubeadm

当前 kubelet 版本 1.7.6

  1. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
  2. cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
  3. deb http://apt.kubernetes.io/ kubernetes-xenial main
  4. EOF
  5. apt-get update
  6. apt-get install -y kubelet kubeadm
  7. # 安装 bash 自动提示
  8. kubeadm completion bash >/etc/profile.d/kubeadm.sh
  9. kubectl completion bash >/etc/profile.d/kubectl.sh
  10. source /etc/profile

初始化 master

  1. kubeadm init --pod-network-cidr=

复制并保存 init token
查看 pod 状态

  1. kubectl get pods --all-namespaces

查看 node 状态

  1. kubectl get nodes


  1. kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看 pod 状态

  1. kubectl get pods --all-namespaces

查看 node 状态

  1. kubectl get nodes


  1. kubeadm join --token $init_token $apiserver-advertise-address:6443

查看 node 状态

  1. kubectl get nodes

部署 helloworld

  1. kubectl apply -f https://raw.githubusercontent.com/jolestar/kubernetes-complete-course/master/example/helloworld.yaml

查看 pod 状态

  1. kubectl get pods

测试 pod 之间以及 pod 和 apiserver 之间的网络

  1. kubectl exec $podname -- nping $pod2ip
  2. kubectl exec $podname -- curl -k https://kubernetes

解决 flannel 网络问题

  1. iptables -P FORWARD ACCEPT
  1. https://github.com/coreos/flannel/issues/799
  2. https://docs.docker.com/engine/userguide/networking/default_network/container-communication/#container-communication-between-hosts

    For security reasons, Docker configures the iptables rules to prevent containers from forwarding traffic from outside the host machine, on Linux hosts. Docker sets the default policy of the FORWARD chain to DROP
    Note: In Docker 1.12 and earlier, the default FORWARD chain policy was ACCEPT. When you upgrade to Docker 1.13 or higher, this default is automatically changed for you

重新测试 pod 之间以及 pod 和 apiserver 之间的网络
关于 Kubernetes 网络以及网络故障的排查,将在后面的 Kubernetes 网络课程里介绍

Kubernetes 的组件

第二课:Kubernetes 的安装和运维 - 图2
  1. kubelet
  2. kube-controller-manager
  3. kube-scheduler
  4. kube-apiserver
  5. kube-proxy
  6. kube-dns
  7. etcd
  8. flannel


  1. 启动方式 系统进程
  2. 配置文件
    • /lib/systemd/system/kubelet.service
    • /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
  3. 主要参数
    • kubeconfig bootstrap-kubeconfig
    • pod-manifest-path
    • allow-privileged(host-network-sources,host-pid-sources,host-ipc-sources)(file,http,api)
    • network-plugin
    • authorization-mod (Webhook、AlwaysAllow)
    • cluster-dns= —cluster-domain=cluster.local
    • feature-gates

Kube-controller-manager 和 kube-scheduler

  1. 启动方式 StaticPod 或 系统进程
  2. 配置文件
    • /etc/kubernetes/manifests/kube-controller-manager.yaml kube-scheduler.yaml
  3. 主要参数(kube-controller-manager)
    • kubeconfig
    • allocate-node-cidrs
    • cluster-cidr=
    • service-cluster-ip-range=
    • leader-elect
    • feature-gates


  1. 启动方式 StaticPod 或 系统进程
  2. 配置文件
    • /etc/kubernetes/manifests/kube-apiserver.yaml
  3. 主要参数
    • kubeconfig
    • insecure-port insecure-bind-address
    • allow-privileged
    • kubelet-preferred-address-types=InternalIP,Hostname
    • authorization-mode=Node,RBAC
    • etcd-servers
    • experimental-bootstrap-token-auth=true
    • service-cluster-ip-range=
    • feature-gates


  1. 启动方式 系统进程 或 DaemonSet
  2. 配置文件
    • kubernetes ds yaml
  3. 主要参数
    • kubeconfig
    • masquerade-all
    • feature-gates


  1. Accelerators=true|false (ALPHA - default=false)
  2. AdvancedAuditing=true|false (ALPHA - default=false)
  3. AffinityInAnnotations=true|false (ALPHA - default=false)
  4. AllAlpha=true|false (ALPHA - default=false)
  5. AllowExtTrafficLocalEndpoints=true|false (default=true)
  6. AppArmor=true|false (BETA - default=true)
  7. DynamicKubeletConfig=true|false (ALPHA - default=false)
  8. DynamicVolumeProvisioning=true|false (ALPHA - default=true)
  9. ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
  10. ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
  11. LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
  12. PersistentLocalVolumes=true|false (ALPHA - default=false)
  13. RotateKubeletClientCertificate=true|false (ALPHA - default=false)
  14. RotateKubeletServerCertificate=true|false (ALPHA - default=false)
  15. StreamingProxyRedirects=true|false (BETA - default=true)
  16. TaintBasedEvictions=true|false (ALPHA - default=false)


  1. 启动方式 Deployment
  2. 配置文件
    • kubernetes deployment yaml
  3. 主要参数(参看配置文件)


  1. 启动方式 StaticPod 或 外部集群
  2. 配置文件
    • /etc/kubernetes/manifests/etcd.yaml
  3. 主要参数(参看配置文件)


  1. 启动方式 StaticPod
  2. 作用 确保系统组件一直存在
  3. 配置文件

Kubernetes 高可用

第二课:Kubernetes 的安装和运维 - 图3


Kubernetes 高可用

  1. Etcd — Cluster
  2. Apiserver — LoadBalancer
  3. kube-controller-manager kube-scheduler — Master elected

Kubernetes selfhosting

kubeadm selfhosting.go 源码

  1. Load the Static Pod specification from disk (from /etc/kubernetes/manifests)
  2. Extract the PodSpec from that Static Pod specification
  3. Mutate the PodSpec to be compatible with self-hosting (add the right labels, taints, etc. so it can schedule correctly)
  4. Build a new DaemonSet object for the self-hosted component in question. Use the above mentioned PodSpec
  5. Create the DaemonSet resource. Wait until the Pods are running.
  6. Remove the Static Pod manifest file. The kubelet will stop the original Static Pod-hosted component that was running.
  7. The self-hosted containers should now step up and take over.
  8. In order to avoid race conditions, we’re still making sure the API /healthz endpoint is healthy
  9. Do that for the kube-apiserver, kube-controller-manager and kube-scheduler in a loop

Kubernetes selfhosting

  1. 更新 kubeadm
    1. wget https://k8s-qingcloud.pek3a.qingstor.com/k8s%2Fv1.7.4%2Fbin%2Fkubeadm -O /usr/bin/kubeadm
  2. selfhosting
    1. kubeadm alpha phase selfhosting
    2. kubectl get pods -n kube-system
    3. kubectl get ds -n kube-system
    4. kubeadm alpha phase mark-master $node
    5. kubectl get pods -n kube-system
    6. kubectl get ds -n kube-system

Kops 和 Minikube

  1. Kops
  2. Minikube https://github.com/kubernetes/minikube

Kubernetes 使用以及运维

  1. kubectl
  2. kubectl proxy
  3. node 相关操作


  1. Basic Commands (Beginner):
  2. create Create a resource by filename or stdin
  3. run Run a particular image on the cluster
  4. set Set specific features on objects (image/resource/selector/subject)
  5. get Display one or many resources
  6. edit Edit a resource on the server
  7. delete Delete resources by filenames, stdin, resources and names, or by resources and label selector
  8. Deploy Commands:
  9. rollout Manage the rollout of a resource
  10. rollingupdate Perform a rolling update of the given ReplicationController
  11. scale Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job
  12. resize Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job
  13. autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController
  14. Cluster Management Commands:
  15. certificate Modify certificate resources.
  16. clusterinfo Display cluster info
  17. top Display Resource (CPU/Memory/Storage) usage.
  18. cordon Mark node as unschedulable
  19. uncordon Mark node as schedulable
  20. drain Drain node in preparation for maintenance
  21. taint Update the taints on one or more nodes


  1. Troubleshooting and Debugging Commands:
  2. describe Show details of a specific resource or group of resources
  3. logs Print the logs for a container in a pod
  4. attach Attach to a running container
  5. exec Execute a command in a container
  6. port-forward Forward one or more local ports to a pod
  7. proxy Run a proxy to the Kubernetes API server
  8. cp Copy files and directories to and from containers.
  9. auth Inspect authorization
  10. Advanced Commands:
  11. apply Apply a configuration to a resource by filename or stdin
  12. patch Update field(s) of a resource using strategic merge patch
  13. replace Replace a resource by filename or stdin
  14. update Replace a resource by filename or stdin
  15. convert Convert config files between different API versions
  16. Settings Commands:
  17. label Update the labels on a resource
  18. annotate Update the annotations on a resource

Kubectl proxy

安装 dashboard

  1. kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

通过 kubectl proxy 在本地查看

  1. kubectl proxy


  1. cordon Mark node as unschedulable
  2. uncordon Mark node as schedulable
  3. drain Drain node in preparation for maintenance
  4. taint Update the taints on one or more nodes


Kubernetes 安装本身并不复杂,但是

  1. 镜像 (gcr.io/quay.io)
  2. 网络
  3. 安全
  4. 扩展插件管理
  5. 配置变更
  6. 集群的伸缩
  7. HA
  8. 升级


  1. 手动通过 kubeadm 搭建一个 Kubernetes 集群,然后安装 kube-addon-manager,通过 kube-addon-manager 管理系统组件,比如 dashboard, kube-proxy, kubedns, heapster 等。
  2. 在本地通过 minikube 搭建一个 Kubernetes 开发集群。
  3. 通过云服务商部署一个托管 Kubernetes 集群,(推荐通过 QingCloud青云的 appcenter 进行部署。https://appcenter.qingcloud.com/apps/app-u0llx5j8) 研究 Kubernetes 的 CloudProvider 机制,以及网络和存储方案。


个人博客: http://jolestar.com
课程 Github:https://github.com/jolestar/kubernetes-complete-course


第二课:Kubernetes 的安装和运维 - 图5