Driver Toolkit

Learn about the Driver Toolkit and how you can use it as a base image for driver containers for enabling special software and hardware devices on Kubernetes.

The Driver Toolkit is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

About the Driver Toolkit

Background

The Driver Toolkit is a container image in the OKD payload used as a base image on which you can build driver containers. The Driver Toolkit image contains the kernel packages commonly required as dependencies to build or install kernel modules, as well as a few tools needed in driver containers. The version of these packages will match the kernel version running on the Fedora CoreOS (FCOS) nodes in the corresponding OKD release.

Driver containers are container images used for building and deploying out-of-tree kernel modules and drivers on container operating systems like FCOS. Kernel modules and drivers are software libraries running with a high level of privilege in the operating system kernel. They extend the kernel functionalities or provide the hardware-specific code required to control new devices. Examples include hardware devices like Field Programmable Gate Arrays (FPGA) or GPUs, and software-defined storage (SDS) solutions, such as Lustre parallel file systems, which require kernel modules on client machines. Driver containers are the first layer of the software stack used to enable these technologies on Kubernetes.

The list of kernel packages in the Driver Toolkit includes the following and their dependencies:

  • kernel-core

  • kernel-devel

  • kernel-headers

  • kernel-modules

  • kernel-modules-extra

In addition, the Driver Toolkit also includes the corresponding real-time kernel packages:

  • kernel-rt-core

  • kernel-rt-devel

  • kernel-rt-modules

  • kernel-rt-modules-extra

The Driver Toolkit also has several tools which are commonly needed to build and install kernel modules, including:

  • elfutils-libelf-devel

  • kmod

  • binutilskabi-dw

  • kernel-abi-whitelists

  • dependencies for the above

Purpose

Prior to the Driver Toolkit’s existence, you could install kernel packages in a pod or build config on OKD using entitled builds or by installing from the kernel RPMs in the hosts machine-os-content. The Driver Toolkit simplifies the process by removing the entitlement step, and avoids the privileged operation of accessing the machine-os-content in a pod. The Driver Toolkit can also be used by partners who have access to pre-released OKD versions to prebuild driver-containers for their hardware devices for future OKD releases.

The Driver Toolkit is also used by the Special Resource Operator (SRO), which is currently available as a community Operator on OperatorHub. SRO supports out-of-tree and third-party kernel drivers and the support software for the underlying operating system. Users can create recipes for SRO to build and deploy a driver container, as well as support software like a device plug-in, or metrics. Recipes can include a build config to build a driver container based on the Driver Toolkit, or SRO can deploy a prebuilt driver container.

Pulling the Driver Toolkit container image

The driver-toolkit image is available from the Container images section of the Red Hat Ecosystem Catalog and in the OKD release payload. The image corresponding to the most recent minor release of OKD will be tagged with the version number in the catalog. The image URL for a specific release can be found using the oc adm CLI command.

Pulling the Driver Toolkit container image from registry.redhat.io

Instructions for pulling the driver-toolkit image from registry.redhat.io with podman or in OKD can be found on the Red Hat Ecosystem Catalog. The driver-toolkit image for the latest minor release will be tagged with the minor release version on registry.redhat.io for example registry.redhat.io/openshift4/driver-toolkit-rhel8:v4.9.

Finding the Driver Toolkit image URL in the payload

Prerequisites

  • You obtained the image pull secret needed to perform an installation of OKD, from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.

  • You installed the OpenShift CLI (oc).

Procedure

  1. The image URL of the driver-toolkit corresponding to a certain release can be extracted from the release image using the oc adm command:

    1. $ oc adm release info 4.9.0 --image-for=driver-toolkit

    Example output

    1. quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd84aee79606178b6561ac71f8540f404d518ae5deff45f6d6ac8f02636c7f4

  2. This image can be pulled using a valid pull secret, such as the pull secret required to install OKD.

  1. $ podman pull --authfile=path/to/pullsecret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<SHA>

Using the Driver Toolkit

As an example, the Driver Toolkit can be used as the base image for building a very simple kernel module called simple-kmod.

The Driver Toolkit contains the necessary dependencies, openssl, mokutil, and keyutils, needed to sign a kernel module. However, in this example, the simple-kmod kernel module is not signed and therefore cannot be loaded on systems with Secure Boot enabled.

Build and run the simple-kmod driver container on a cluster

Prerequisites

  • You have a running OKD cluster.

  • You set the Image Registry Operator state to Managed for your cluster.

  • You installed the OpenShift CLI (oc).

  • You are logged into the OpenShift CLI as a user with cluster-admin privileges.

Procedure

Create a namespace. For example:

  1. $ oc new-project simple-kmod-demo

  1. The YAML defines an ImageStream for storing the simple-kmod driver container image, and a BuildConfig for building the container. Save this YAML as 0000-buildconfig.yaml.template.

    1. apiVersion: image.openshift.io/v1
    2. kind: ImageStream
    3. metadata:
    4.   labels:
    5.     app: simple-kmod-driver-container
    6.   name: simple-kmod-driver-container
    7.   namespace: simple-kmod-demo
    8. spec: {}
    9. ---
    10. apiVersion: build.openshift.io/v1
    11. kind: BuildConfig
    12. metadata:
    13.   labels:
    14.     app: simple-kmod-driver-build
    15.   name: simple-kmod-driver-build
    16.   namespace: simple-kmod-demo
    17. spec:
    18.   nodeSelector:
    19.     node-role.kubernetes.io/worker: ""
    20.   runPolicy: "Serial"
    21.   triggers:
    22.     - type: "ConfigChange"
    23.     - type: "ImageChange"
    24.   source:
    25.     git:
    26.       ref: "master"
    27.       uri: "https://github.com/openshift-psap/kvc-simple-kmod.git"
    28.     type: Git
    29.     dockerfile: |
    30.       FROM DRIVER_TOOLKIT_IMAGE
    32.       WORKDIR /build/
    34.       RUN yum -y install git make sudo gcc \
    35.       && yum clean all \
    36.       && rm -rf /var/cache/dnf
    38.       # Expecting kmod software version as an input to the build
    39.       ARG KMODVER
    41.       # Grab the software from upstream
    42.       RUN git clone https://github.com/openshift-psap/simple-kmod.git
    43.       WORKDIR simple-kmod
    45.       # Prep and build the module
    46.       RUN make buildprep KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}"  kernel-core) KMODVER=${KMODVER} \
    47.       && make all       KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}"  kernel-core) KMODVER=${KMODVER} \
    48.       && make install   KVER=$(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}"  kernel-core) KMODVER=${KMODVER}
    50.       # Add the helper tools
    51.       WORKDIR /root/kvc-simple-kmod
    52.       ADD Makefile .
    53.       ADD simple-kmod-lib.sh .
    54.       ADD simple-kmod-wrapper.sh .
    55.       ADD simple-kmod.conf .
    56.       RUN mkdir -p /usr/lib/kvc/ \
    57.       && mkdir -p /etc/kvc/ \
    58.       && make install
    60.       RUN systemctl enable kmods-via-containers@simple-kmod
    61.   strategy:
    62.     dockerStrategy:
    63.       buildArgs:
    64.         - name: KMODVER
    65.           value: DEMO
    66.   output:
    67.     to:
    68.       kind: ImageStreamTag
    69.       name: simple-kmod-driver-container:demo

  2. Substitute the correct driver toolkit image for the OKD version you are running in place of “DRIVER_TOOLKIT_IMAGE” with the following commands.

    1. $ OCP_VERSION=$(oc get clusterversion/version -ojsonpath={.status.desired.version})
    1. $ DRIVER_TOOLKIT_IMAGE=$(oc adm release info $OCP_VERSION --image-for=driver-toolkit)
    1. $ sed "s#DRIVER_TOOLKIT_IMAGE#${DRIVER_TOOLKIT_IMAGE}#" 0000-buildconfig.yaml.template > 0000-buildconfig.yaml

  3. Create the image stream and build config with

    1. $ oc create -f 0000-buildconfig.yaml

  4. After the builder pod completes successfully, deploy the driver container image as a DaemonSet.

    1. The driver container must run with the privileged security context in order to load the kernel modules on the host. The following YAML file contains the RBAC rules and the DaemonSet for running the driver container. Save this YAML as 1000-drivercontainer.yaml.

      1. apiVersion: v1
      2. kind: ServiceAccount
      3. metadata:
      4.   name: simple-kmod-driver-container
      5. ---
      6. apiVersion: rbac.authorization.k8s.io/v1
      7. kind: Role
      8. metadata:
      9.   name: simple-kmod-driver-container
      10. rules:
      11. - apiGroups:
      12.   - security.openshift.io
      13.   resources:
      14.   - securitycontextconstraints
      15.   verbs:
      16.   - use
      17.   resourceNames:
      18.   - privileged
      19. ---
      20. apiVersion: rbac.authorization.k8s.io/v1
      21. kind: RoleBinding
      22. metadata:
      23.   name: simple-kmod-driver-container
      24. roleRef:
      25.   apiGroup: rbac.authorization.k8s.io
      26.   kind: Role
      27.   name: simple-kmod-driver-container
      28. subjects:
      29. - kind: ServiceAccount
      30.   name: simple-kmod-driver-container
      31. userNames:
      32. - system:serviceaccount:simple-kmod-demo:simple-kmod-driver-container
      33. ---
      34. apiVersion: apps/v1
      35. kind: DaemonSet
      36. metadata:
      37.   name: simple-kmod-driver-container
      38. spec:
      39.   selector:
      40.     matchLabels:
      41.       app: simple-kmod-driver-container
      42.   template:
      43.     metadata:
      44.       labels:
      45.         app: simple-kmod-driver-container
      46.     spec:
      47.       serviceAccount: simple-kmod-driver-container
      48.       serviceAccountName: simple-kmod-driver-container
      49.       containers:
      50.       - image: image-registry.openshift-image-registry.svc:5000/simple-kmod-demo/simple-kmod-driver-container:demo
      51.         name: simple-kmod-driver-container
      52.         imagePullPolicy: Always
      53.         command: ["/sbin/init"]
      54.         lifecycle:
      55.           preStop:
      56.             exec:
      57.               command: ["/bin/sh", "-c", "systemctl stop kmods-via-containers@simple-kmod"]
      58.         securityContext:
      59.           privileged: true
      60.       nodeSelector:
      61.         node-role.kubernetes.io/worker: ""

    2. Create the RBAC rules and daemon set:

      1. $ oc create -f 1000-drivercontainer.yaml

  5. After the pods are running on the worker nodes, verify that the simple_kmod kernel module is loaded successfully on the host machines with lsmod.

    1. Verify that the pods are running:

      1. $ oc get pod -n simple-kmod-demo

      Example output

      1. NAME                                 READY   STATUS      RESTARTS   AGE
      2. simple-kmod-driver-build-1-build     0/1     Completed   0          6m
      3. simple-kmod-driver-container-b22fd   1/1     Running     0          40s
      4. simple-kmod-driver-container-jz9vn   1/1     Running     0          40s
      5. simple-kmod-driver-container-p45cc   1/1     Running     0          40s

    2. Execute the lsmod command in the driver container pod:

      1. $ oc exec -it pod/simple-kmod-driver-container-p45cc -- lsmod | grep simple

      Example output

      1. simple_procfs_kmod     16384  0
      2. simple_kmod            16384  0

Additional resources