How to bootstrap an edge Kubernetes cluster

1. Background

At present, many edge computing container open source projects have a default prerequisite for use: users need to prepare a standard or specific tool to build a Kubernetes cluster in advance, and then use specific tools or other methods to deploy in the cluster deployment corresponding components to experience edge capabilities. This undoubtedly raises the threshold of user experience edge capabilities, and there are many restrictions on use, making it difficult for users to get started. Simply organize, there will probably be the following problems:

  • The threshold is too high

    • Users need to prepare a Kubernetes cluster in advance. For many users, the threshold is too high, the construction process is more complicated and easy to fail, and many people who want to use edge capabilities are shut out;
  • Too restrictive

    • Often require specific tools to build a specific version of Kubernetes cluster, the versatility is too poor, users want to use too restrictive in the actual production environment;
  • Adding edge nodes is more troublesome

    • Adding edge nodes needs to rely on the tools to build the Kubernetes cluster itself to add Kubernetes native nodes and then convert, which is highly dependent on third-party tools, and the operation process is more troublesome and error-prone;
  • Poor automation

    • Regardless of the establishment of a Kubernetes cluster or the addition of edge nodes, it is difficult to automate in the production environment. The related processes also require secondary development by their own team, which is difficult to integrate;

2. Architecture Design

In response to the above problems, in order to lower the threshold for users to use the edge Kubernetes cluster and make the edge Kubernetes cluster capable of production, we designed a one-click solution to deploy an edge Kubernetes cluster, completely shielding the installation details, so that users can have a zero-threshold experience Edge capacity.

2.1 Original intention

  • It is very simple for users to use edge Kubernetes clusters without barriers, and can truly use edge capabilities in the production environment;

2.2 Goal

  • One-click

    • Able to build an edge Kubernetes cluster with one click;
    • It is very simple and flexible to add edge nodes with one key;
  • Two kinds of installation creation

    • Support online installation;
    • Support offline installation, so that the privatization environment can also be very simple;
  • Support online installation;

    • Support offline installation, so that the privatization environment can also be very simple;
  • Can be used in production

    • Don’t encapsulate too much, so that teams who want to use the edge Kubernetes cluster can perform simple integration in the internal system, and then it is available for production;
  • Zero learning cost

    • Try to be consistent with the use of kubeadm as much as possible, so that users will use kubeadm and edgeadm without additional learning costs;

2.3 Principle

  • Don’t modify kubeadm source code
    • Try to quote and reuse the source code of kubeadm, and try not to modify the source code of kubeadm to avoid hidden dangers of subsequent upgrades;
    • Based on kubeadm but higher than kubeadm, it does not have to be limited by the design of kubeadm, it can be allowed as long as it makes it easier for users to use;
  • Allow users to choose whether to deploy edge capability components;
  • Allow users to customize the configuration of edge capability components;

2.4 Design and implementation


We studied the source code of Kubeadm and found that we can borrow Kubeadm to create native Kubernetes clusters, join nodes, and workflow ideas to deploy edge Kubernetes clusters with one click, and perform the installation steps step by step. This is exactly what we want for a simple, flexible, and low learning cost deployment solution. So we stood on the shoulders of giants, used Kubedam’s ideas, reused Kubeadm’s source code, and designed the following solution.

Bootstrap edge Kubernetes cluster - 图1

Among them, the part of Kubeadm init cluster/join node completely reuses the source code of kubadm, and all the logic is exactly the same as Kubeadm.

This program has the following advantages:

  • Fully compatible with Kubeadm

    We just stand on the shoulders of Kubeadm, set some configuration parameters required by the edge cluster before Kubeadm init/join, initialize the Master or Node nodes automatically, and install the container runtime. After the completion of Kubeadm init/join, the CNI network plug-in was installed and the corresponding edge capability components were deployed.

    We quoted the Kubeadm source code in the way of Go Mod. During the whole process, we did not modify the source code of Kubeadm one line. It is completely native and ready to upgrade to a higher version of Kubeadm in the future.

  • One-click, easy to use, flexible and automated

    The edgeadm init cluster and join node completely retain the original parameters and process of Kubeadm init/join, but automatically initialize the node and install the container when running, you can use the edgeadm --enable-edge=fasle parameter to install the native one-click For Kubernetes clusters, you can also use the edgeadm --enable-edge=true parameter to install an edge Kubernetes cluster with one click.

    You can join any node as long as you can access the node where the Kube-apiserver is located, or you can join the master. Join master also continues the Kubeadm approach. To build highly available nodes, you can directly use join master to expand Master nodes when needed to achieve high availability.

  • No learning cost, exactly the same as using kubeadm

    Because the Kubeadm init cluster/join node part completely reuses the source code of kubadm, all logic is exactly the same as Kubeadm, completely retaining the usage habits of kubeadm and all flag parameters, and the usage is exactly the same as that of kubeadm, without any new learning cost , The user can customize the edge Kubernetes cluster according to the parameters of Kubeadm or use kubeadm.config.

  • Edge node security enhancement

    With the help of Kubernetes Node Authentication mechanism, we have enabled NodeRestriction access plugin to ensure that each node has a unique identity and only has a minimal set of permissions. Even if an edge node is compromised, other edge nodes cannot be operated.

    For Kubelet, we also enable the Kubelet configuration certificate rotation mechanism by default. When the Kubelet certificate is about to expire, a new secret key will be automatically generated , And apply for a new certificate from the Kubernetes API. Once the new certificate is available, it will be used to authenticate the connection with the Kubernetes API.

3. Install edge Kubernetes cluster with edgeadm

<1>. Installation conditions

  • Follow kubeadm’s minimum requirements, master && node minimum 2C2G, disk space Not less than 1G;

    Warning: Provide clean machines as much as possible to avoid installation errors caused by other factors. If there is a container service on the machine, it may be cleaned up during the installation process, please confirm carefully before executing it

  • Currently supports amd64 and arm64 two systems;

    Other systems can compile edgeadm and make corresponding system installation packages by themselves, please refer to 5. Customize Kubernetes static installation package

  • Supported Kubernetes version: greater than or equal to v1.18, the provided installation package only provides Kubernetes v1.18.2 version;

    For other Kubernetes versions, please refer to 5. Customize the Kubernetes static installation package and make it yourself.

<2>. Download the edgeadm static installation package and copy it to all master && node nodes

Choose installation package according to your installation node CPU architecture [amd64, arm64]

  1. arch=amd64 version=v0.3.0 && rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version && ./edgeadm

The installation package is about 200M. For detailed information about the installation package, please refer to 5. Custom Kubernetes static installation package.

If downloading the installation package is slow, you can directly check the corresponding SuperEdge version, download edgeadm-linux-amd64/arm64-*.0.tgz, and Decompression is the same.

One-click installation of the edge independent Kubernetes cluster function is supported starting from SuperEdge-v0.3.0-beta.0, pay attention to

<3>. Install edge Kubernetes master node

  1. ./edgeadm init --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --install-pkg-path ./kube-linux-*.tar.gz --apiserver-cert-extra-sans=<Master public IP> --apiserver-advertise-address=<master Intranet IP> --enable-edge=true -v=6

On:

  • –enable-edge=true: Whether to deploy edge capability components, the default is true

    –enable-edge=false means to install a native Kubernetes cluster, which is exactly the same as the cluster built by kubeadm;

  • –install-pkg-path: The address of the Kubernetes static installation package

    The value of –install-pkg-path can be the path on the machine or the network address (for example: http://xxx/xxx/kube-linux-arm64/amd64-\*.tar.gz, which can be encrypted without wget You can), pay attention to use the Kubernetes static installation package that matches the machine system;

  • –apiserver-cert-extra-sans: kube-apiserver certificate extension address

    • It is recommended to sign the master node’s external network IP or domain name, as long as the signed Master node’s IP or domain name can be accessed by the edge node**. Of course, the internal network IP is also allowed, provided that the edge node can access kube-apiserver through this IP. If you customize the domain name, you can configure hosts on all Matser and Node nodes by yourself;
    • Sign the public network IP and domain name, because the edge node and the Master node are generally not in the same local area network, and need to join and access the Master through the public network;
  • –image-repository: image repository address

    If superedge.tencentcloudcr.com/superedge is slower, you can switch to other accelerated mirror warehouses, as long as you can pull down kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, etcd, pause, etc. mirrors.

Other parameters have the same meaning as Kubeadm and can be configured according to kubeadm’s requirements.

You can also use kubeadm.config to configure the original parameters of kubeadm, and create an edge Kubernetes cluster through edgeadm init --config kubeadm.config --install-pkg-path ./kube-linux-*.tar.gz .

If there is no problem during execution and the cluster is successfully initialized, the following content will be output:

  1. Your Kubernetes control-plane has initialized successfully!
  2. To start using your cluster, you need to run the following as a regular user:
  3. mkdir -p $HOME/.kube
  4. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  5. sudo chown $(id -u):$(id -g) $HOME/.kube/config
  6. You should now deploy a pod network to the cluster.
  7. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  8. https://kubernetes.io/docs/concepts/cluster-administration/addons/
  9. Then you can join any number of worker nodes by running the following on each as root:
  10. edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
  11. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
  12. --install-pkg-path <Path of edgeadm kube-* install package>

If there is a problem during the execution, the corresponding error message will be returned directly and the initialization of the cluster will be interrupted. You can use the ./edgeadm reset command to roll back the initialization operation of the cluster.

<4>. Set the master kube-config file

To enable non-root users to run kubectl, run the following commands, which are also part of the edgeadm init output:

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you are the root user, you can run:

  1. export KUBECONFIG=/etc/kubernetes/admin.conf

Note that the ./edgeadm join command that saves the output of ./edgeadm init will be used when adding node nodes later.

The validity period of the token is the same as kubeadm 24h, after expiration, you can use ./edgeadm token create to create a new token.

The value generation of –discovery-token-ca-cert-hash is also the same as kubeadm, which can be generated by executing the following command on the master node.

  1. openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

<5>. Join edge nodes

Execute <2>. Download the edgeadm static installation package on the edge node, or upload the edgeadm static installation package to the edge node by other means, and then execute the following command:

  1. ./edgeadm join <Master public/Intranet IP or domain>:Port --token xxxx \
  2. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
  3. --install-pkg-path <edgeadm Kube-*Static installation package address/FTP path>
  4. --enable-edge=true

On:

  • <Master public/Intranet IP or domain>: the address where the node accesses the Kube-apiserver service.

    You can change the address of the Kube-apiserver service prompted by the edgeadm init to the node to be replaced by Master node public network IP/Master node internal network IP/domain name depending on the situation, depending on whether you want the node to access Kube through the external network or the internal network -apiserver service.

  • –enable-edge=true: Whether the added node is used as an edge node (whether to deploy edge capability components), the default is true

    –enable-edge=false means join the native Kubernetes cluster node, which is exactly the same as the node joined by kubeadm;

If there are no exceptions in the execution process, the new node successfully joins the cluster, and the following will be output:

  1. This node has joined the cluster:
  2. * Certificate signing request was sent to apiserver and a response was received.
  3. * The Kubelet was informed of the new secure connection details.
  4. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

If there is a problem during the execution, the corresponding error message will be returned directly, and the addition of the node will be interrupted. You can use the ./edgeadm reset command to roll back the operation of joining the node and rejoin.

Tip: If the edge node is joined, after the edge node joins successfully, the edge node will be labeled with a label: superedge.io/edge-node=enable, which is convenient for subsequent applications to use nodeSelector to select the application and schedule to the edge node;

Native Kubernetes nodes, like kubeadm’s join, do not do anything.

4. Use edgeadm to install edge high-availability Kubernetes cluster

<1>. Installation prerequisites

<2>. Install Haproxy

Install Haproxy on the load balancing machine as the main entrance of the cluster:

Note: Replace in the configuration file

  1. # yum install -y haproxy
  2. # cat << EOF >/etc/haproxy/haproxy.cfg
  3. global
  4. log 127.0.0.1 local2
  5. chroot /var/lib/haproxy
  6. pidfile /var/run/haproxy.pid
  7. maxconn 4000
  8. user haproxy
  9. group haproxy
  10. daemon
  11. stats socket /var/lib/haproxy/stats
  12. defaults
  13. mode http
  14. log global
  15. option httplog
  16. option dontlognull
  17. option http-server-close
  18. option forwardfor except 127.0.0.0/8
  19. option redispatch
  20. retries 3
  21. timeout http-request 10s
  22. timeout queue 1m
  23. timeout connect 10s
  24. timeout client 1m
  25. timeout server 1m
  26. timeout http-keep-alive 10s
  27. timeout check 10s
  28. maxconn 3000
  29. frontend main *:5000
  30. acl url_static path_beg -i /static /images /javascript /stylesheets
  31. acl url_static path_end -i .jpg .gif .png .css .js
  32. use_backend static if url_static
  33. default_backend app
  34. frontend kubernetes-apiserver
  35. mode tcp
  36. bind *:16443
  37. option tcplog
  38. default_backend kubernetes-apiserver
  39. backend kubernetes-apiserver
  40. mode tcp
  41. balance roundrobin
  42. server master-0 <master VIP>:6443 check # Here replace the master VIP with the user's own VIP
  43. backend static
  44. balance roundrobin
  45. server static 127.0.0.1:4331 check
  46. backend app
  47. balance roundrobin
  48. server app1 127.0.0.1:5001 check
  49. server app2 127.0.0.1:5002 check
  50. server app3 127.0.0.1:5003 check
  51. server app4 127.0.0.1:5004 check
  52. EOF

<3>. Install Keepalived

If the cluster has two masters, install Keepalived on both masters and perform the same operation:

Note:

  1. Replace in the configuration file

  2. In the keepalived.conf configuration file below, <master’s local public network IP> and <another master’s public network IP> are opposite in the configuration of the two masters. Don’t fill in the error.

  1. # yum install -y keepalived
  2. # cat << EOF >/etc/keepalived/keepalived.conf
  3. ! Configuration File for keepalived
  4. global_defs {
  5. smtp_connect_timeout 30
  6. router_id LVS_DEVEL_EDGE_1
  7. }
  8. vrrp_script checkhaproxy{
  9. script "/etc/keepalived/do_sth.sh"
  10. interval 5
  11. }
  12. vrrp_instance VI_1 {
  13. state BACKUP
  14. interface eth0
  15. nopreempt
  16. virtual_router_id 51
  17. priority 100
  18. advert_int 1
  19. authentication {
  20. auth_type PASS
  21. auth_pass aaa
  22. }
  23. virtual_ipaddress {
  24. <master VIP> # Here replace the master VIP with the user's own VIP
  25. }
  26. unicast_src_ip <master Public IP>
  27. unicast_peer {
  28. <Public IP of other master nodes>
  29. }
  30. notify_master "/etc/keepalived/notify_action.sh MASTER"
  31. notify_backup "/etc/keepalived/notify_action.sh BACKUP"
  32. notify_fault "/etc/keepalived/notify_action.sh FAULT"
  33. notify_stop "/etc/keepalived/notify_action.sh STOP"
  34. garp_master_delay 1
  35. garp_master_refresh 5
  36. track_interface {
  37. eth0
  38. }
  39. track_script {
  40. checkhaproxy
  41. }
  42. }
  43. EOF

<4>. Install high-availability edge Kubernetes master

Perform cluster initialization operations in one of the masters

  1. ./edgeadm init --control-plane-endpoint <Master VIP> --upload-certs --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans=<Domain or Public/Intranet IP of Master node> --install-pkg-path <edgeadm Kube-*Static installation package address/FTP path> -v=6

The meaning of the parameters is the same as 3. Use edgeadm to install edge Kubernetes cluster, and others are the same as kubeadm, so I won’t explain it here;

If there are no exceptions during execution and the cluster is successfully initialized, the following content will be output:

  1. Your Kubernetes control-plane has initialized successfully!
  2. To start using your cluster, you need to run the following as a regular user:
  3. mkdir -p $HOME/.kube
  4. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  5. sudo chown $(id -u):$(id -g) $HOME/.kube/config
  6. You should now deploy a pod network to the cluster.
  7. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  8. https://kubernetes.io/docs/concepts/cluster-administration/addons/
  9. You can now join any number of the control-plane node running the following command on each as root:
  10. edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
  11. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \
  12. --control-plane --certificate-key xxxxxxxxxx
  13. --install-pkg-path <Path of edgeadm kube-* install package>
  14. Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
  15. As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
  16. "edgeadm init phase upload-certs --upload-certs" to reload certs afterward.
  17. Then you can join any number of worker nodes by running the following on each as root:
  18. edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \
  19. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
  20. --install-pkg-path <Path of edgeadm kube-* install package>

If there is a problem during the execution, the corresponding error message will be directly returned, and the initialization of the cluster will be interrupted. Use the ./edgeadm reset command to roll back the initialization operation of the cluster.

To enable non-root users to run kubectl, run the following commands, which are also part of the edgeadm init output:

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you are the root user, you can run:

  1. export KUBECONFIG=/etc/kubernetes/admin.conf

Pay attention to the ./edgeadm join command that saves the output of ./edgeadm init, which is needed to add Master node and edge node later.

Record the ./edgeadm join command output by ./edgeadm init. You need this command to add the Master node and the edge node.

<5>. Join master node

Execute the ./edgeadm join command on another master

  1. ./edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
  2. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \
  3. --control-plane --certificate-key xxxxxxxxxx \
  4. --install-pkg-path <Path of edgeadm kube-* install package>

If there are no exceptions in the execution process, the new master successfully joins the cluster, and the following content will be output:

  1. This node has joined the cluster and a new control plane instance was created:
  2. * Certificate signing request was sent to apiserver and approval was received.
  3. * The Kubelet was informed of the new secure connection details.
  4. * Control plane (master) label and taint were applied to the new node.
  5. * The Kubernetes control plane instances scaled up.
  6. * A new etcd member was added to the local/stacked etcd cluster.
  7. To start administering your cluster from this node, you need to run the following as a regular user:
  8. mkdir -p $HOME/.kube
  9. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  10. sudo chown $(id -u):$(id -g) $HOME/.kube/config
  11. Run 'kubectl get nodes' to see this node join the cluster.

If there is a problem during the execution, the corresponding error message will be directly returned, and the addition of the node will be interrupted. Use the ./edgeadm reset command to roll back the initialization of the cluster.

<6>. Join edge node

  1. ./edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \
  2. --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
  3. --install-pkg-path <Path of edgeadm kube-* install package>

If there are no exceptions in the execution process, the new master successfully joins the cluster, and the following content will be output:

  1. This node has joined the cluster:
  2. * Certificate signing request was sent to apiserver and a response was received.
  3. * The Kubelet was informed of the new secure connection details.
  4. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

If there are exceptions during execution, the corresponding error message will be directly returned, and the initialization of the cluster will be interrupted. Use the ./edgeadm reset command to roll back the initialization operation of the cluster.

5. Customize the Kubernetes static installation package

  1. kube-linux-arm64-v1.18.2.tar.gz ## Kubernetes static installation package for kube-v1.18.2
  2. ├── bin ## Binary directory
  3. ├── conntrack ## Binary file for connection tracking
  4. ├── kubectl ## kubectl for kube-v1.18.2
  5. ├── kubelet ## kubelet for kube-v1.18.2
  6. └── lite-apiserver ## The corresponding version of lite-apiserver
  7. ├── cni ## cin configuration
  8. └── cni-plugins-linux-v0.8.3.tar.gz ## CNI plug-in binary compression package of v0.8.3
  9. └── container ## Container runtime directory
  10. └── docker-19.03-linux-arm64.tar.gz ## Docker 19.03 arm64 system installation script and installation package

<1>. Customize other Kubernetes versions

There are two things you need to do to customize other Kubernetes versions:

  • Replace the kubectl and kubelet files in the binary directory, the version needs to be greater than or equal to Kubernetes v1.18.0;
  • Ensure that there is a basic image of the corresponding Kubernetes version in the mirror warehouse used by init;

<2>. Customize other system Kubernetes static installation packages

To customize the Kubernetes static installation package and other systems, three things need to be done:

  • Replace all the binaries of the Kubernetes static installation package with the target system, including the binaries in the corresponding installation packages of cni and container;
  • Make sure that the mirror warehouse used by init has the basic image of the Kubernetes version of the corresponding system. It is recommended to use Multi-System Mirror;
  • Fully test to ensure that there are no compatibility issues. If there are related problems, you can also mention Issues in the SuperEdge community to fix them.

Last modified June 15, 2021 : initial commit (974355a)