Prerequisites (iSCSI client)

OpenEBS configuration flow

OpenEBS provides block volume support through the iSCSI protocol. Therefore, the iSCSI client (initiator) presence on all Kubernetes nodes is required. Choose the platform below to find the steps to verify if the iSCSI client is installed and running or to find the steps to install the iSCSI client.

Choose the platform for iSCSI client settings

Prerequisites - 图2Ubuntu

Prerequisites - 图3RHEL

Prerequisites - 图4CentOS

Prerequisites - 图5EKS

Prerequisites - 图6GKE

Prerequisites - 图7AKS

Prerequisites - 图8OpenShift

Prerequisites - 图9Rancher

Prerequisites - 图10ICP

Prerequisites - 图11 DigitalOcean

Provide feedback if a platform is missing in the above list.

Linux platforms

Installation of the iSCSI initiator service and tools depends on your host O/S or the kubelet container. You can follow the below steps for installation/verification of the required packages. It is a mandatory step to verify the iSCSI services and make sure that it is running on all the worker nodes. OpenEBS uses the iSCSI protocol to connect to the block volumes.

Ubuntu

Verify iSCSI services are configured

If an iSCSI initiator is already installed on your node, check that the initiator name is configured and iSCSI service is running using the following commands.

  1. sudo cat /etc/iscsi/initiatorname.iscsi
  2. systemctl status iscsid

If service status is showing as Inactive, then you may have to enable and start the iscsid service using the following command.

  1. sudo systemctl enable iscsid && sudo systemctl start iscsid

The following is the expected output.

  1. systemctl status iscsid
  2. ● iscsid.service - iSCSI initiator daemon (iscsid)
  3.    Loaded: loaded (/lib/systemd/system/iscsid.service; disabled; vendor preset: enabled)
  4.    Active: active (running) since Mon 2019-02-18 11:00:07 UTC; 1min 51s ago
  5.      Docs: man:iscsid(8)
  6.   Process: 11185 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
  7.   Process: 11170 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
  8.  Main PID: 11187 (iscsid)
  9.     Tasks: 2 (limit: 4915)
  10.    CGroup: /system.slice/iscsid.service
  11.            ├─11186 /sbin/iscsid
  12.            └─11187 /sbin/iscsid

Install the iSCSI tools

If an iSCSI initiator is not installed on your node, install open-iscsi packages using the following commands.

  1. sudo apt-get update
  2. sudo apt-get install open-iscsi
  3. sudo service open-iscsi restart

You can verify the iSCSI installation from above section.

Red Hat Enterprise Linux

Verify iSCSI services are configured

In Red Hat Enterprise Linux 7, the iSCSI service is lazily started by default: the service starts after running the iscsiadm command. If an iSCSI initiator is already installed on the node, check that the initiator name is configured using the following command.

  1.  cat /etc/iscsi/initiatorname.iscsi

Check iSCSI service is running using the following command.

  1.  systemctl status iscsid

If the status is showing as Inactive, then you may have to enable and start the iscsid service using the following command.

  1. sudo systemctl enable iscsid && sudo systemctl start iscsid

The following is the expected output.

  1. systemctl status iscsid
  2. ● iscsid.service - Open-iSCSI
  3.    Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
  4.    Active: active (running) since Tue 2019-02-19 12:19:08 IST; 2h 37min ago
  5.      Docs: man:iscsid(8)
  6.            man:iscsiadm(8)
  7.  Main PID: 2138 (iscsid)
  8.     Tasks: 2
  9.    CGroup: /system.slice/iscsid.service
  10.            ├─2137 /usr/sbin/iscsid
  11.            └─2138 /usr/sbin/iscsid
  13. Feb 19 12:19:08 master-1550555590.mayalab.com systemd[1]: Starting Open-iSCSI...
  14. Feb 19 12:19:08 master-1550555590.mayalab.com iscsid[2136]: iSCSI logger with pid=2137 started!
  15. Feb 19 12:19:08 master-1550555590.mayalab.com systemd[1]: Failed to read PID from file /var/run/iscsid.pid: Invalid argument
  16. Feb 19 12:19:08 master-1550555590.mayalab.com systemd[1]: Started Open-iSCSI.
  17. Feb 19 12:19:09 master-1550555590.mayalab.com iscsid[2137]: iSCSI daemon with pid=2138 started!

Install the iSCSI tools

If an iSCSI initiator is not installed on your node, install iscsi-initiator-utils packages using the following command.

  1. yum install iscsi-initiator-utils -y

You can verify the iSCSI installation from above section.

CentOS

Verify iSCSI services are configured

If an iSCSI initiator is already installed on your node, check that the initiator name is configured using the following commands.

  1.  cat /etc/iscsi/initiatorname.iscsi

Check iSCSI service is running using the following command.

  1.  systemctl status iscsid

If the status is showing as Inactive, then you may have to enable and start the iscsid service using the following command.

  1. sudo systemctl enable iscsid && sudo systemctl start iscsid

Install the iSCSI tools

If an iSCSI initiator is not installed on your node, install iscsi-initiator-utils packages using the following command.

  1. yum install iscsi-initiator-utils -y

You can verify the iSCSI installation from the above section.

Managed Kubernetes Services on Public Cloud

Amazon Elastic Container Service for Kubernetes (EKS)

Amazon EKS clusters can be brought up with either an AmazonLinux AMI or an Ubuntu 18.04 AMI.

For clusters running with the AmazonLinux AMI

Verify iSCSI services are configured

If an iSCSI initiator is already installed on your node, check that the initiator name is configured using the following commands.

  1.  cat /etc/iscsi/initiatorname.iscsi

Check the iSCSI service is running using the following command.

  1.  systemctl status iscsid

If the status is showing as Inactive, then you may have to enable and start the iscsid service using the following command.

  1. sudo systemctl enable iscsid && sudo systemctl start iscsid

Install the iSCSI tools

If an iSCSI initiator is not installed on your node, install iscsi-initiator-utils packages using the following command.

  1. yum install iscsi-initiator-utils -y

You can verify the iSCSI installation from the above section.

For clusters running with the Ubuntu 18.04 AMI

For setting up iSCSI clients on Ubuntu nodes, see the instructions here.

Google Kubernetes Engine (GKE)

GKE Container-Optimized OS does not come with an iSCSI client preinstalled and does not allow installation of an iSCSI client. Therefore, OpenEBS does not work on Kubernetes clusters which are running GKE Container-Optimized OS version of the image on the worker nodes.

Select Ubuntu as the image version for the node pools in the custom settings. For setting up iSCSI clients on Ubuntu nodes, see the instructions here.

Azure Kubernetes Service (AKS)

On Azure Kubernetes Service you need to verify that the open-iscsi packages are installed and running the service on the kubelet. This can be checked by connecting to the nodes through SSH using the public IP addresses by running the following command.

  1. ssh azureuser@40.xx.yyy.zzz

Note: azureuser is a default username.

Verify iSCSI services are configured

Obtain the container ID of the hyperkube kubelet on each node using the following command.

  1. sudo docker ps | grep "hyperkube kubelet"

Following is the example output:

  1. 3aab0f9a48e2    k8s-gcrio.azureedge.net/hyperkube-amd64:v1.8.7   "/hyperkube kubele..."   48 minutes ago      Up 48 minutes                           eager_einstein

Once kubelet container ID is obtained, you need to get to the shell of this container using the following command.

  1. sudo docker exec -it <container ID> bash

Example:

  1. sudo docker exec -it 3aab0f9a48e2 bash

Check the status of the iSCSI service by using the following command.

  1. service open-iscsi status

Install the iSCSI tools

You have to get the kubelet container ID using the steps mentioned in the above section. Once kubelet container ID is obtained, you need to get to the shell of this container using the following command.

  1. sudo docker exec -it <container ID> bash

Example:

  1. sudo docker exec -it 3aab0f9a48e2 bash

Run the following commands to install and configure iSCSI service in the kubelet.

  1. apt-get update
  2. apt install -y open-iscsi
  3. exit

You can verify the iSCSI installation from the above section.

DigitalOcean

Add extra_binds in Kubelet Service

Add the following lines (volume mounts) to the file /etc/systemd/system/kubelet.service on each of the nodes: 

  1. -v /sbin/iscsiadm:/usr/bin/iscsiadm \
  2. -v /lib/x86_64-linux-gnu/libisns-nocrypto.so.0:/lib/x86_64-linux-gnu/libisns-nocrypto.so.0 \

So, the updated Kubelet Service File is as below:

  1. [Unit]
  2. Description=Kubernetes Kubelet Server
  3. Documentation=https://kubernetes.io/docs/concepts/overview/components/#kubelet
  4. After=docker.service sys-fs-bpf.mount
  5. Requires=docker.service sys-fs-bpf.mount
  6. [Service]
  7. OOMScoreAdjust=-999
  8. ExecStartPre=/bin/mkdir -p /var/lib/kubelet
  9. ExecStartPre=/bin/mount — bind /var/lib/kubelet /var/lib/kubelet
  10. ExecStartPre=/bin/mount — make-shared /var/lib/kubelet
  11. ExecStart=/usr/bin/docker run — rm — net=host — pid=host — privileged — name kubelet \
  12. -v /dev:/dev \
  13. -v /sys:/sys \
  14. -v /var:/var \
  15. -v /var/lib/kubelet:/var/lib/kubelet:shared \
  16. -v /etc:/etc \
  17. -v /run:/run \
  18. -v /opt:/opt \
  19. -v /sbin/iscsiadm:/usr/bin/iscsiadm \
  20. -v /lib/x86_64-linux-gnu/libisns-nocrypto.so.0:/lib/x86_64-linux-gnu/libisns-nocrypto.so.0 \
  21. gcr.io/google-containers/hyperkube:v1.15.3 \
  22. /hyperkube kubelet \
  23. — config=/etc/kubernetes/kubelet.conf \
  24. — feature-gates=”RuntimeClass=false” \
  25. — logtostderr=true \
  26. — image-pull-progress-deadline=2m \
  27. — kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  28. — bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  29. — rotate-certificates \
  30. — register-node=true \
  31. — node-labels=”doks.digitalocean.com/node-id=32559d91-cc04–4aac-bdc4–0566fa066802,doks.digitalocean.com/node-pool-id=d5714f37–627d-435a-b1c7-f0373ecd7593,doks.digitalocean.com/node-pool=pool-nuyzam6e8,doks.digitalocean.com/version=1.15.3-do.2" \
  32. — root-dir=/var/lib/kubelet \
  33. — v=2 \
  34. — cloud-provider=external \
  35. — network-plugin=cni \
  36. — provider-id=”digitalocean://160254521"
  37. Restart=on-failure
  38. RestartSec=5
  39. KillMode=process
  40. [Install]
  41. WantedBy=multi-user.target

Next, you need to restart the Kubelet Service on each node using the following commands

  1. systemctl daemon-reload
  2. service kubelet restart

Kubernetes On-Prem Solutions

Red Hat OpenShift

OpenEBS can be installed using Openshift on the following Operating Systems. Tested OpenShift versions are 3.9 and 3.10.

  1. On RHEL 7
  2. On CentOS 7

On RHEL

Latest tested RHEL versions are 7.5, 7.6.

For setting up iSCSI clients on RHEL nodes, see the instructions here.

On CentOS

Latest tested CentOS version is CentOS 7.

For setting up iSCSI clients on CentOS nodes, see the instructions here.

Rancher

  • If you are using RancherOS as the operating system, you need to enable the iSCSI service and start it on all the worker nodes.
  • If you are using Ubuntu or RHEL as the operating system, you need to
    • Verify if iSCSI initiators are installed on all nodes (and )
    • Add the extra_binds under Kubelet service in cluster YAML file to mount the iSCSI binary and configuration inside the kubelet.

iSCSI services On RancherOS

To run iSCSI services, execute the following commands on each of the cluster hosts or nodes.

  1. sudo ros s enable open-iscsi
  2. sudo ros s up open-iscsi

Run the below commands on all the nodes to make sure the below directories are persistent, by default these directories are ephemeral.

  1. ros config set rancher.services.user-volumes.volumes  [/home:/home,/opt:/opt,/var/lib/kubelet:/var/lib/kubelet,/etc/kubernetes:/etc/kubernetes,/var/openebs]
  2. system-docker rm all-volumes
  3. reboot

If you are using Jiva or Local PV for provisioning OpenEBS volume on hostpath, add default hostpath of corresponding storage engine to extra_binds under kubelet service in cluster YAML. If the volume is using a mounted path on the host, then you must add the mounted path under extra_binds section.

  1. services:
  2.   kubelet:
  3.     extra_binds:
  4.      - /var/openebs/local:/var/openebs/local

In the above snippet, default hostpath for Local PV (/var/openebs/local), which will be created on the worker node using openebs-hostpath StorageClass, is added under extra_binds. This configuration will help to create default hostpath directory on worker node for provisioning openebs-hostpath volume.

iSCSI services on RHEL or Ubuntu 16.04

Step1: Verify iSCSI initiator is installed and services are running

Operating systemiSCSI PackageCommands
RHEL / CentOSiscsi-initiator-utilsyum install iscsi-initiator-utils -y
sudo systemctl enable iscsid && sudo systemctl start iscsid
modprobe iscsi_tcp
echo iscsi_tcp >/etc/modules-load.d/iscsi-tcp.conf
Ubuntu 16.04 / Debianopen-iscsisudo apt install open-iscsi
sudo systemctl enable iscsid && sudo systemctl start iscsid
modprobe iscsi_tcp
echo iscsi_tcp >/etc/modules-load.d/iscsi-tcp.conf

Step2: Add extra_binds under kubelet service in cluster YAML

After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below.

  1. services:
  2.     kubelet: 
  3.       extra_binds: 
  4.         - "/etc/iscsi:/etc/iscsi"
  5.         - "/sbin/iscsiadm:/sbin/iscsiadm"
  6.         - "/var/lib/iscsi:/var/lib/iscsi"
  7.         - "/lib/modules"

iSCSI services on Ubuntu 18.04 or CentOS 7.6

Step1: By default, iSCSI service is not present on worker node. It will be running inside the kubelet. To verify presence of iSCSI service inside kubelet, run the following command:

  1. docker exec kubelet iscsiadm -V

Example Output:

  1. iscsiadm version 2.0-874

The following commands will enable the iscsi_tcp module and it will persist this changes to the system.

Operating systemiSCSI PackageCommands
Ubuntu 18.04open-iscsimodprobe iscsi_tcp
echo iscsi_tcp >/etc/modules-load.d/iscsi-tcp.conf
CentOS 7.6iscsi-initiator-utilsmodprobe iscsi_tcp
echo iscsi_tcp >/etc/modules-load.d/iscsi-tcp.conf

Step 2: If you are using Jiva or Local PV for provisioning OpenEBS volume on hostpath, add default hostpath of corresponding storage engine to extra_binds under kubelet service in cluster YAML. If the volume is using a mounted path on the host, then you must add the mounted path under extra_binds section.

  1. services:
  2.   kubelet:
  3.     extra_binds:
  4.      - /var/openebs/local:/var/openebs/local

In the above snippet, default hostpath for Local PV (/var/openebs/local), which will be created on the worker node using openebs-hostpath StorageClass, is added under extra_binds. This configuration will help to create default hostpath directory on worker node for provisioning openebs-hostpath volume.

IBM Cloud Private (ICP)

OpenEBS can be installed using ICP on the following Operating Systems. Latest tested ICP versions are 2.1.0.3 and 3.1.1.

  1. On RHEL 7

  2. On Ubuntu

On RHEL

Latest tested RHEL versions are 7.5, 7.6.

For setting up iSCSI clients on RHEL nodes, see the instructions here.

On Ubuntu

Latest tested Ubuntu version are Ubuntu 16.04 LTS and 18.04 LTS.

For setting up iSCSI clients on Ubuntu nodes, see the instructions here.

See Also:

OpenEBS Installation

OpenEBS Architecture