Bootstrapping the Kubernetes Worker Nodes

In this lab you will bootstrap three Kubernetes worker nodes. The following components will be installed on each node: runc, container networking plugins, containerd, kubelet, and kube-proxy.

Prerequisites

The commands in this lab must be run on each worker instance: worker-0, worker-1, and worker-2. Login to each worker instance using the gcloud command. Example:

  1. gcloud compute ssh worker-0

Running commands in parallel with tmux

tmux can be used to run commands on multiple compute instances at the same time. See the Running commands in parallel with tmux section in the Prerequisites lab.

Provisioning a Kubernetes Worker Node

Install the OS dependencies:

  1. {
  2. sudo apt-get update
  3. sudo apt-get -y install socat conntrack ipset
  4. }

The socat binary enables support for the kubectl port-forward command.

Disable Swap

By default the kubelet will fail to start if swap is enabled. It is recommended that swap be disabled to ensure Kubernetes can provide proper resource allocation and quality of service.

Verify if swap is enabled:

  1. sudo swapon --show

If output is empthy then swap is not enabled. If swap is enabled run the following command to disable swap immediately:

  1. sudo swapoff -a

To ensure swap remains off after reboot consult your Linux distro documentation.

Download and Install Worker Binaries

  1. wget -q --show-progress --https-only --timestamping \
  2. https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz \
  3. https://github.com/opencontainers/runc/releases/download/v1.0.0-rc93/runc.amd64 \
  4. https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz \
  5. https://github.com/containerd/containerd/releases/download/v1.4.4/containerd-1.4.4-linux-amd64.tar.gz \
  6. https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kubectl \
  7. https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kube-proxy \
  8. https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/amd64/kubelet

Create the installation directories:

  1. sudo mkdir -p \
  2. /etc/cni/net.d \
  3. /opt/cni/bin \
  4. /var/lib/kubelet \
  5. /var/lib/kube-proxy \
  6. /var/lib/kubernetes \
  7. /var/run/kubernetes

Install the worker binaries:

  1. {
  2. mkdir containerd
  3. tar -xvf crictl-v1.21.0-linux-amd64.tar.gz
  4. tar -xvf containerd-1.4.4-linux-amd64.tar.gz -C containerd
  5. sudo tar -xvf cni-plugins-linux-amd64-v0.9.1.tgz -C /opt/cni/bin/
  6. sudo mv runc.amd64 runc
  7. chmod +x crictl kubectl kube-proxy kubelet runc
  8. sudo mv crictl kubectl kube-proxy kubelet runc /usr/local/bin/
  9. sudo mv containerd/bin/* /bin/
  10. }

Configure CNI Networking

Retrieve the Pod CIDR range for the current compute instance:

  1. POD_CIDR=$(curl -s -H "Metadata-Flavor: Google" \
  2. http://metadata.google.internal/computeMetadata/v1/instance/attributes/pod-cidr)

Create the bridge network configuration file:

  1. cat <<EOF | sudo tee /etc/cni/net.d/10-bridge.conf
  2. {
  3. "cniVersion": "0.4.0",
  4. "name": "bridge",
  5. "type": "bridge",
  6. "bridge": "cnio0",
  7. "isGateway": true,
  8. "ipMasq": true,
  9. "ipam": {
  10. "type": "host-local",
  11. "ranges": [
  12. [{"subnet": "${POD_CIDR}"}]
  13. ],
  14. "routes": [{"dst": "0.0.0.0/0"}]
  15. }
  16. }
  17. EOF

Create the loopback network configuration file:

  1. cat <<EOF | sudo tee /etc/cni/net.d/99-loopback.conf
  2. {
  3. "cniVersion": "0.4.0",
  4. "name": "lo",
  5. "type": "loopback"
  6. }
  7. EOF

Configure containerd

Create the containerd configuration file:

  1. sudo mkdir -p /etc/containerd/
  1. cat << EOF | sudo tee /etc/containerd/config.toml
  2. [plugins]
  3. [plugins.cri.containerd]
  4. snapshotter = "overlayfs"
  5. [plugins.cri.containerd.default_runtime]
  6. runtime_type = "io.containerd.runtime.v1.linux"
  7. runtime_engine = "/usr/local/bin/runc"
  8. runtime_root = ""
  9. EOF

Create the containerd.service systemd unit file:

  1. cat <<EOF | sudo tee /etc/systemd/system/containerd.service
  2. [Unit]
  3. Description=containerd container runtime
  4. Documentation=https://containerd.io
  5. After=network.target
  6. [Service]
  7. ExecStartPre=/sbin/modprobe overlay
  8. ExecStart=/bin/containerd
  9. Restart=always
  10. RestartSec=5
  11. Delegate=yes
  12. KillMode=process
  13. OOMScoreAdjust=-999
  14. LimitNOFILE=1048576
  15. LimitNPROC=infinity
  16. LimitCORE=infinity
  17. [Install]
  18. WantedBy=multi-user.target
  19. EOF

Configure the Kubelet

  1. {
  2. sudo mv ${HOSTNAME}-key.pem ${HOSTNAME}.pem /var/lib/kubelet/
  3. sudo mv ${HOSTNAME}.kubeconfig /var/lib/kubelet/kubeconfig
  4. sudo mv ca.pem /var/lib/kubernetes/
  5. }

Create the kubelet-config.yaml configuration file:

  1. cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
  2. kind: KubeletConfiguration
  3. apiVersion: kubelet.config.k8s.io/v1beta1
  4. authentication:
  5. anonymous:
  6. enabled: false
  7. webhook:
  8. enabled: true
  9. x509:
  10. clientCAFile: "/var/lib/kubernetes/ca.pem"
  11. authorization:
  12. mode: Webhook
  13. clusterDomain: "cluster.local"
  14. clusterDNS:
  15. - "10.32.0.10"
  16. podCIDR: "${POD_CIDR}"
  17. resolvConf: "/run/systemd/resolve/resolv.conf"
  18. runtimeRequestTimeout: "15m"
  19. tlsCertFile: "/var/lib/kubelet/${HOSTNAME}.pem"
  20. tlsPrivateKeyFile: "/var/lib/kubelet/${HOSTNAME}-key.pem"
  21. EOF

The resolvConf configuration is used to avoid loops when using CoreDNS for service discovery on systems running systemd-resolved.

Create the kubelet.service systemd unit file:

  1. cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
  2. [Unit]
  3. Description=Kubernetes Kubelet
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=containerd.service
  6. Requires=containerd.service
  7. [Service]
  8. ExecStart=/usr/local/bin/kubelet \\
  9. --config=/var/lib/kubelet/kubelet-config.yaml \\
  10. --container-runtime=remote \\
  11. --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
  12. --image-pull-progress-deadline=2m \\
  13. --kubeconfig=/var/lib/kubelet/kubeconfig \\
  14. --network-plugin=cni \\
  15. --register-node=true \\
  16. --v=2
  17. Restart=on-failure
  18. RestartSec=5
  19. [Install]
  20. WantedBy=multi-user.target
  21. EOF

Configure the Kubernetes Proxy

  1. sudo mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig

Create the kube-proxy-config.yaml configuration file:

  1. cat <<EOF | sudo tee /var/lib/kube-proxy/kube-proxy-config.yaml
  2. kind: KubeProxyConfiguration
  3. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  4. clientConnection:
  5. kubeconfig: "/var/lib/kube-proxy/kubeconfig"
  6. mode: "iptables"
  7. clusterCIDR: "10.200.0.0/16"
  8. EOF

Create the kube-proxy.service systemd unit file:

  1. cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
  2. [Unit]
  3. Description=Kubernetes Kube Proxy
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. [Service]
  6. ExecStart=/usr/local/bin/kube-proxy \\
  7. --config=/var/lib/kube-proxy/kube-proxy-config.yaml
  8. Restart=on-failure
  9. RestartSec=5
  10. [Install]
  11. WantedBy=multi-user.target
  12. EOF

Start the Worker Services

  1. {
  2. sudo systemctl daemon-reload
  3. sudo systemctl enable containerd kubelet kube-proxy
  4. sudo systemctl start containerd kubelet kube-proxy
  5. }

Remember to run the above commands on each worker node: worker-0, worker-1, and worker-2.

Verification

The compute instances created in this tutorial will not have permission to complete this section. Run the following commands from the same machine used to create the compute instances.

List the registered Kubernetes nodes:

  1. gcloud compute ssh controller-0 \
  2. --command "kubectl get nodes --kubeconfig admin.kubeconfig"

output

  1. NAME STATUS ROLES AGE VERSION
  2. worker-0 Ready <none> 22s v1.21.0
  3. worker-1 Ready <none> 22s v1.21.0
  4. worker-2 Ready <none> 22s v1.21.0

Next: Configuring kubectl for Remote Access