Complete Example Using GlusterFS for Dynamic Provisioning

Overview

This topic provides an end-to-end example of how to use an existing Containerized GlusterFS, External GlusterFS, or standalone GlusterFS cluster as dynamic persistent storage for OKD. It is assumed that a working GlusterFS cluster is already set up. For help installing Containerized GlusterFS or External GlusterFS, see Persistent Storage Using GlusterFS.

All oc commands are executed on the OKD master host.

Prerequisites

To access GlusterFS volumes, the mount.glusterfs command must be available on all schedulable nodes. For RPM-based systems, the glusterfs-fuse package must be installed:

  1. # yum install glusterfs-fuse

If glusterfs-fuse is already installed on the nodes, ensure that the latest version is installed:

  1. # yum update glusterfs-fuse

By default, SELinux does not allow writing from a pod to a remote GlusterFS server. To enable writing to GlusterFS volumes with SELinux on, run the following on each node running GlusterFS:

  1. $ sudo setsebool -P virt_sandbox_use_fusefs on (1)
  2. $ sudo setsebool -P virt_use_fusefs on
1The -P option makes the boolean persistent between reboots.

The virt_sandbox_use_fusefs boolean is defined by the docker-selinux package. If you get an error saying it is not defined, ensure that this package is installed.

If you use Atomic Host, the SELinux booleans are cleared when you upgrade Atomic Host. When you upgrade Atomic Host, you must set these boolean values again.

Dynamic Provisioning

  1. To enable dynamic provisioning, first create a StorageClass object definition. The definition below is based on the minimum requirements needed for this example to work with OKD. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4. name: glusterfs
    5. provisioner: kubernetes.io/glusterfs
    6. parameters:
    7. resturl: "http://10.42.0.0:8080" (1)
    8. restauthenabled: "false" (2)
    1The heketi server URL.
    2Since authentication is not turned on in this example, set to false.
  2. From the OKD master host, create the StorageClass:

    1. # oc create -f gluster-storage-class.yaml
    2. storageclass "glusterfs" created
  3. Create a PVC using the newly-created StorageClass. For example:

    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4. name: gluster1
    5. spec:
    6. accessModes:
    7. - ReadWriteMany
    8. resources:
    9. requests:
    10. storage: 30Gi
    11. storageClassName: glusterfs
  4. From the OKD master host, create the PVC:

    1. # oc create -f glusterfs-dyn-pvc.yaml
    2. persistentvolumeclaim "gluster1" created
  5. View the PVC to see that the volume was dynamically created and bound to the PVC:

    1. # oc get pvc
    2. NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
    3. gluster1 Bound pvc-78852230-d8e2-11e6-a3fa-0800279cf26f 30Gi RWX glusterfs 42s

Using the Storage

At this point, you have a dynamically created GlusterFS volume bound to a PVC. You can now utilize this PVC in a pod.

  1. Create the pod object definition:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. name: hello-openshift-pod
    5. labels:
    6. name: hello-openshift-pod
    7. spec:
    8. containers:
    9. - name: hello-openshift-pod
    10. image: openshift/hello-openshift
    11. ports:
    12. - name: web
    13. containerPort: 80
    14. volumeMounts:
    15. - name: gluster-vol1
    16. mountPath: /usr/share/nginx/html
    17. readOnly: false
    18. volumes:
    19. - name: gluster-vol1
    20. persistentVolumeClaim:
    21. claimName: gluster1 (1)
    1The name of the PVC created in the previous step.
  2. From the OKD master host, create the pod:

    1. # oc create -f hello-openshift-pod.yaml
    2. pod "hello-openshift-pod" created
  3. View the pod. Give it a few minutes, as it might need to download the image if it does not already exist:

    1. # oc get pods -o wide
    2. NAME READY STATUS RESTARTS AGE IP NODE
    3. hello-openshift-pod 1/1 Running 0 9m 10.38.0.0 node1
  4. oc exec into the container and create an index.html file in the mountPath definition of the pod:

    1. $ oc exec -ti hello-openshift-pod /bin/sh
    2. $ cd /usr/share/nginx/html
    3. $ echo 'Hello OpenShift!!!' > index.html
    4. $ ls
    5. index.html
    6. $ exit
  5. Now curl the URL of the pod:

    1. # curl http://10.38.0.0
    2. Hello OpenShift!!!
  6. Delete the pod, recreate it, and wait for it to come up:

    1. # oc delete pod hello-openshift-pod
    2. pod "hello-openshift-pod" deleted
    3. # oc create -f hello-openshift-pod.yaml
    4. pod "hello-openshift-pod" created
    5. # oc get pods -o wide
    6. NAME READY STATUS RESTARTS AGE IP NODE
    7. hello-openshift-pod 1/1 Running 0 9m 10.37.0.0 node1
  7. Now curl the pod again and it should still have the same data as before. Note that its IP address may have changed:

    1. # curl http://10.37.0.0
    2. Hello OpenShift!!!
  8. Check that the index.html file was written to GlusterFS storage by doing the following on any of the nodes:

    1. $ mount | grep heketi
    2. /dev/mapper/VolGroup00-LogVol00 on /var/lib/heketi type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    3. /dev/mapper/vg_f92e09091f6b20ab12b02a2513e4ed90-brick_1e730a5462c352835055018e1874e578 on /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_1e730a5462c352835055018e1874e578 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
    4. /dev/mapper/vg_f92e09091f6b20ab12b02a2513e4ed90-brick_d8c06e606ff4cc29ccb9d018c73ee292 on /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_d8c06e606ff4cc29ccb9d018c73ee292 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
    5. $ cd /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_d8c06e606ff4cc29ccb9d018c73ee292/brick
    6. $ ls
    7. index.html
    8. $ cat index.html
    9. Hello OpenShift!!!