KubeEdge runtime configuration

containerd

Docker 18.09 ships with containerd, so you needn’t install containerd manually, otherwise install containerd in this way,

  1. # Install containerd
  2. apt-get update && apt-get install -y containerd.io
  4. # Configure containerd
  5. mkdir -p /etc/containerd
  6. containerd config default > /etc/containerd/config.toml
  8. # Restart containerd
  9. systemctl restart containerd

In case of using containerd shipped with Docker, you still need to update containerd’s configuration, this is because “cri” plugin is disabled by default, you need to enable it so that KubeEdge can use the containerd as the runtime.

  1. # Configure containerd
  2. mkdir -p /etc/containerd
  3. containerd config default > /etc/containerd/config.toml

Update edgecore config file edgecore.yaml, specify the following parameters for containerd based runtime,

  1. remoteRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
  2. remoteImageEndpoint: unix:///var/run/containerd/containerd.sock
  3. runtimeRequestTimeout: 2
  4. podSandboxImage: k8s.gcr.io/pause:3.2
  5. runtimeType: remote

By default, cgroup driver of CRI is configured as cgroupfs, if this is not the case, you can switch to systemd manually in edgecore.yaml,

  1. modules:
  2.   edged:
  3.     cgroupDriver: systemd

Set systemd_cgroup to true in containerd’s configuration file (/etc/containerd/config.toml ), restart contained service after that.

  1. # /etc/containerd/config.toml
  2. systemd_cgroup = true
  1. # Restart containerd
  2. systemctl restart containerd

Create the nginx application and check the container is created with containerd on edge side,

  1. kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml
  2. deployment.apps/nginx-deployment created
  4. ctr --namespace=k8s.io container ls
  5. CONTAINER                                                           IMAGE                              RUNTIME
  6. 41c1a07fe7bf7425094a9b3be285c312127961c158f30fc308fd6a3b7376eab2    docker.io/library/nginx:1.15.12    io.containerd.runtime.v1.linux

NOTE: since cri doesn’t support multi-tenancy while containerd does, namespace for containers are set to “k8s.io” defaultly and no way to change that until the cri’s support has done.

CRI-O

Follow the CRI-O install guide to setup CRI-O.

If your edge node is running on ARM platform and your distro is ubuntu18.04, you might need to build the binaries form source and install after that, since CRI-O packages is not available in Kubic repository for this combination.

  1. git clone https://github.com/cri-o/cri-o
  2. cd cri-o
  3. make
  4. sudo make install
  5. # generate and install configuraion files
  6. sudo make install.config

Setup CNI networking, please follow this guide setup CNI to set up the CNI networking. Update edgecore config file, specify the following parameters for CRI-O based runtime,

  1. remoteRuntimeEndpoint: unix:///var/run/crio/crio.sock
  2. remoteImageEndpoint: unix:////var/run/crio/crio.sock
  3. runtimeRequestTimeout: 2
  4. podSandboxImage: k8s.gcr.io/pause:3.2
  5. runtimeType: remote

By default, CRI-O uses cgroupfs as a cgroup driver manager, update the CRI-O config file (/etc/crio/crio.conf.d/00-default.conf) like this if you want to switch to systemd instead.

  1. # Cgroup management implementation used for the runtime.
  2. cgroup_manager = "systemd"

NOTE: pause image should be updated if you are on ARM platform and the pause image you are using is not a multi-arch image.

Here is an example, update the CRI-O config file to make the pause image points to the correct image,

  1. pause_image = "k8s.gcr.io/pause-arm64:3.1"

Remember to update edgecore.yaml as well for your cgroup driver manager,

  1. modules:
  2.   edged:
  3.     cgroupDriver: systemd

Start CRI-O and edgecore services (assume both services are taken care of by systemd),

  1. sudo systemctl daemon-reload
  2. sudo systemctl enable crio
  3. sudo systemctl start crio
  4. sudo systemctl start edgecore

Create the application and check the container is created with CRI-O on edge side,

  1. kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml
  2. deployment.apps/nginx-deployment created
  4. # crictl ps
  5. CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
  6. 41c1a07fe7bf7       f6d22dec9931b       2 days ago          Running             nginx               0                   51f727498b06f

Kata-container

Kata-container is created to primarily address the security challenges in the multi-tenant untrusted cloud environment, multi-tenancy support is still in KubeEdge’s backlog. If you have a downstream customized KubeEdge which supports multi-tenancy already then Kata-container as a lightweight and secure container runtime is good option for you.

Follow the install guide to install and configure containerd and Kata Containers.

If you have “kata-runtime” installed, run this command to check if your host system can run and create a Kata Container,

  1. kata-runtime kata-check

RuntimeClass is a feature for selecting the container runtime configuration to use to run a pod’s containers that is supported since containerd v1.2.0, thus, if your containerd is later than v1.2.0, you will have two choices to configure containerd to use Kata Containers, “Kata Containers as a RuntimeClass” or “Kata Containers as the runtime for untrusted workload”. Suppose you have configured Kata Containers as the runtime for untrusted workload, here is the way you can verify whether it works on your edge node,

  1. cat nginx-untrusted.yaml
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5.   name: nginx-untrusted
  6.   annotations:
  7.     io.kubernetes.cri.untrusted-workload: "true"
  8. spec:
  9.   containers:
  10.   - name: nginx
  11.     image: nginx
  1. kubectl create -f nginx-untrusted.yaml
  3. # verify the container is running with qemu hypervisor on edge side,
  4. ps aux | grep qemu
  5. root      3941  3.0  1.0 2971576 174648 ?      Sl   17:38   0:02 /usr/bin/qemu-system-aarch64
  7. crictl pods
  8. POD ID              CREATED              STATE               NAME                NAMESPACE           ATTEMPT
  9. b1c0911644cb9       About a minute ago   Ready               nginx-untrusted     default             0

Virtlet

Make sure no libvirt is running on the worker nodes.

Steps

  1. Install CNI plugin:

    • Download CNI plugin release and extract it:
    1. $ wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz
    3. # Extract the tarball
    4. $ mkdir cni
    5. $ tar -zxvf v0.2.0.tar.gz -C cni
    7. $ mkdir -p /opt/cni/bin
    8. $ cp ./cni/* /opt/cni/bin/
    • Configure cni plugin
    1. $ mkdir -p /etc/cni/net.d/
    3. $ cat >/etc/cni/net.d/bridge.conf <<EOF
    4. {
    5.   "cniVersion": "0.3.1",
    6.   "name": "containerd-net",
    7.   "type": "bridge",
    8.   "bridge": "cni0",
    9.   "isGateway": true,
    10.   "ipMasq": true,
    11.   "ipam": {
    12.     "type": "host-local",
    13.     "subnet": "10.88.0.0/16",
    14.     "routes": [
    15.       { "dst": "0.0.0.0/0" }
    16.     ]
    17.   }
    18. }
    19. EOF

  2. Setup VM runtime: Use script hack/setup-vmruntime.sh to set up VM runtime. It makes use of Arktos Runtime release to start three containers:

    1.  vmruntime_vms
    2.  vmruntime_libvirt
    3.  vmruntime_virtlet