Persistent Volume Snapshots

Overview

Persistent Volume Snapshots are a Technology Preview feature.

Many storage systems provide the ability to create “snapshots” of a persistent volume (PV) to protect against data loss. The external snapshot controller and provisioner provide means to use the feature in the OKD cluster and handle volume snapshots through the OKD API.

This document describes the current state of volume snapshot support in OKD. Familiarity with PVs, persistent volume claims (PVCs), and dynamic provisioning is recommended.

Features

  • Create snapshot of a PersistentVolume bound to a PersistentVolumeClaim

  • List existing VolumeSnapshots

  • Delete existing VolumeSnapshot

  • Create a new PersistentVolume from an existing VolumeSnapshot

  • Supported PersistentVolume types:

    • AWS Elastic Block Store (EBS)

    • Google Compute Engine (GCE) Persistent Disk (PD)

Installation and Setup

The external controller and provisioner are the external components that provide volume snapshotting. These external components run in the cluster. The controller is responsible for creating, deleting, and reporting events on volume snapshots. The provisioner creates new PersistentVolumes from the volume snapshots. See Create Snapshot and Restore Snapshot for more information.

Starting the External Controller and Provisioner

The external controller and provisioner services are distributed as container images and can be run in the OKD cluster as usual. There are also RPM versions for the controller and provisioner.

To allow the containers managing the API objects, the necessary role-based access control (RBAC) rules need to be configured by the administrator:

  1. Create a ServiceAccount and ClusterRole:

    1. apiVersion: v1
    2. kind: ServiceAccount
    3. metadata:
    4.   name: snapshot-controller-runner
    5. kind: ClusterRole
    6. apiVersion: rbac.authorization.k8s.io/v1
    7. metadata:
    8.   name: snapshot-controller-role
    9. rules:
    10.   - apiGroups: [""]
    11.     resources: ["persistentvolumes"]
    12.     verbs: ["get", "list", "watch", "create", "delete"]
    13.   - apiGroups: [""]
    14.     resources: ["persistentvolumeclaims"]
    15.     verbs: ["get", "list", "watch", "update"]
    16.   - apiGroups: ["storage.k8s.io"]
    17.     resources: ["storageclasses"]
    18.     verbs: ["get", "list", "watch"]
    19.   - apiGroups: [""]
    20.     resources: ["events"]
    21.     verbs: ["list", "watch", "create", "update", "patch"]
    22.   - apiGroups: ["apiextensions.k8s.io"]
    23.     resources: ["customresourcedefinitions"]
    24.     verbs: ["create", "list", "watch", "delete"]
    25.   - apiGroups: ["volumesnapshot.external-storage.k8s.io"]
    26.     resources: ["volumesnapshots"]
    27.     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    28.   - apiGroups: ["volumesnapshot.external-storage.k8s.io"]
    29.     resources: ["volumesnapshotdatas"]
    30.     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

  2. Bind the rules via ClusterRoleBinding:

    1. apiVersion: rbac.authorization.k8s.io/v1beta1
    2. kind: ClusterRoleBinding
    3. metadata:
    4.   name: snapshot-controller
    5. roleRef:
    6.   apiGroup: rbac.authorization.k8s.io
    7.   kind: ClusterRole
    8.   name: snapshot-controller-role
    9. subjects:
    10. - kind: ServiceAccount
    11.   name: snapshot-controller-runner
    12.   namespace: default

If the external controller and provisioner are deployed in Amazon Web Services (AWS), they must be able to authenticate using the access key. To provide the credential to the pod, the administrator creates a new secret:

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: awskeys
  5. type: Opaque
  6. data:
  7.   access-key-id: <base64 encoded AWS_ACCESS_KEY_ID>
  8.   secret-access-key: <base64 encoded AWS_SECRET_ACCESS_KEY>

The AWS deployment of the external controller and provisioner containers (note that both pod containers use the secret to access the AWS cloud provider API):

  1. kind: Deployment
  2. apiVersion: extensions/v1beta1
  3. metadata:
  4.   name: snapshot-controller
  5. spec:
  6.   replicas: 1
  7.   strategy:
  8.     type: Recreate
  9.   template:
  10.     metadata:
  11.       labels:
  12.         app: snapshot-controller
  13.     spec:
  14.       serviceAccountName: snapshot-controller-runner
  15.       containers:
  16.         - name: snapshot-controller
  17.           image: "registry.redhat.io/openshift3/snapshot-controller:latest"
  18.           imagePullPolicy: "IfNotPresent"
  19.           args: ["-cloudprovider", "aws"]
  20.           env:
  21.             - name: AWS_ACCESS_KEY_ID
  22.               valueFrom:
  23.                 secretKeyRef:
  24.                   name: awskeys
  25.                   key: access-key-id
  26.             - name: AWS_SECRET_ACCESS_KEY
  27.               valueFrom:
  28.                 secretKeyRef:
  29.                   name: awskeys
  30.                   key: secret-access-key
  31.         - name: snapshot-provisioner
  32.           image: "registry.redhat.io/openshift3/snapshot-provisioner:latest"
  33.           imagePullPolicy: "IfNotPresent"
  34.           args: ["-cloudprovider", "aws"]
  35.           env:
  36.             - name: AWS_ACCESS_KEY_ID
  37.               valueFrom:
  38.                 secretKeyRef:
  39.                   name: awskeys
  40.                   key: access-key-id
  41.             - name: AWS_SECRET_ACCESS_KEY
  42.               valueFrom:
  43.                 secretKeyRef:
  44.                   name: awskeys
  45.                   key: secret-access-key

For GCE, there is no need to use secrets to access the GCE cloud provider API. The administrator can proceed with the deployment:

  1. kind: Deployment
  2. apiVersion: extensions/v1beta1
  3. metadata:
  4.   name: snapshot-controller
  5. spec:
  6.   replicas: 1
  7.   strategy:
  8.     type: Recreate
  9.   template:
  10.     metadata:
  11.       labels:
  12.         app: snapshot-controller
  13.     spec:
  14.       serviceAccountName: snapshot-controller-runner
  15.       containers:
  16.         - name: snapshot-controller
  17.           image: "registry.redhat.io/openshift3/snapshot-controller:latest"
  18.           imagePullPolicy: "IfNotPresent"
  19.           args: ["-cloudprovider", "gce"]
  20.         - name: snapshot-provisioner
  21.           image: "registry.redhat.io/openshift3/snapshot-provisioner:latest"
  22.           imagePullPolicy: "IfNotPresent"
  23.           args: ["-cloudprovider", "gce"]

Managing Snapshot Users

Depending on the cluster configuration, it might be necessary to allow non-administrator users to manipulate the VolumeSnapshot objects on the API server. This can be done by creating a ClusterRole bound to a particular user or group.

For example, assume the user ‘alice’ needs to work with snapshots in the cluster. The cluster administrator completes the following steps:

  1. Define a new ClusterRole:

    1. apiVersion: v1
    2. kind: ClusterRole
    3. metadata:
    4.   name: volumesnapshot-admin
    5. rules:
    6. - apiGroups:
    7.   - "volumesnapshot.external-storage.k8s.io"
    8.   attributeRestrictions: null
    9.   resources:
    10.   - volumesnapshots
    11.   verbs:
    12.   - create
    13.   - delete
    14.   - deletecollection
    15.   - get
    16.   - list
    17.   - patch
    18.   - update
    19.   - watch

  2. Bind the cluster role to the user ‘alice’ by creating a ClusterRoleBinding object:

    1. apiVersion: rbac.authorization.k8s.io/v1beta1
    2. kind: ClusterRoleBinding
    3. metadata:
    4.   name: volumesnapshot-admin
    5. roleRef:
    6.   apiGroup: rbac.authorization.k8s.io
    7.   kind: ClusterRole
    8.   name: volumesnapshot-admin
    9. subjects:
    10. - kind: User
    11.   name: alice

This is only an example of API access configuration. The VolumeSnapshot objects behave similar to other OKD API objects. See the API access control documentation for more information on managing the API RBAC.

Lifecycle of a Volume Snapshot and Volume Snapshot Data

Persistent Volume Claim and Persistent Volume

The PersistentVolumeClaim is bound to a PersistentVolume. The PersistentVolume type must be one of the snapshot supported persistent volume types.

Snapshot Promoter

To create a StorageClass:

  1. kind: StorageClass
  2. apiVersion: storage.k8s.io/v1
  3. metadata:
  4.   name: snapshot-promoter
  5. provisioner: volumesnapshot.external-storage.k8s.io/snapshot-promoter

This StorageClass is necessary to restore a PersistentVolume from a VolumeSnapshot that was previously created.

Create Snapshot

To take a snapshot of a PV, create a new VolumeSnapshot object:

  1. apiVersion: volumesnapshot.external-storage.k8s.io/v1
  2. kind: VolumeSnapshot
  3. metadata:
  4.   name: snapshot-demo
  5. spec:
  6.   persistentVolumeClaimName: ebs-pvc

persistentVolumeClaimName is the name of the PersistentVolumeClaim bound to a PersistentVolume. This particular PV is snapshotted.

A VolumeSnapshotData object is then automatically created based on the VolumeSnapshot. The relationship between VolumeSnapshot and VolumeSnapshotData is similar to the relationship between PersistentVolumeClaim and PersistentVolume.

Depending on the PV type, the operation might go through several phases, which are reflected by the VolumeSnapshot status:

  1. The new VolumeSnapshot object is created.

  2. The controller starts the snapshot operation. The snapshotted PersistentVolume might need to be frozen and the applications paused.

  3. The storage system finishes creating the snapshot (the snapshot is “cut”) and the snapshotted PersistentVolume might return to normal operation. The snapshot itself is not yet ready. The last status condition is of Pending type with status value True. A new VolumeSnapshotData object is created to represent the actual snapshot.

  4. The newly created snapshot is complete and ready to use. The last status condition is of Ready type with status value True.

It is the user’s responsibility to ensure data consistency (stop the pod/application, flush caches, freeze the file system, and so on).

In case of error, the VolumeSnapshot status is appended with an Error condition.

To display the VolumeSnapshot status:

  1. $ oc get volumesnapshot -o yaml

The status is displayed.

  1. apiVersion: volumesnapshot.external-storage.k8s.io/v1
  2. kind: VolumeSnapshot
  3. metadata:
  4.   clusterName: ""
  5.   creationTimestamp: 2017-09-19T13:58:28Z
  6.   generation: 0
  7.   labels:
  8.     Timestamp: "1505829508178510973"
  9.   name: snapshot-demo
  10.   namespace: default
  11.   resourceVersion: "780"
  12.   selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/snapshot-demo
  13.   uid: 9cc5da57-9d42-11e7-9b25-90b11c132b3f
  14. spec:
  15.   persistentVolumeClaimName: ebs-pvc
  16.   snapshotDataName: k8s-volume-snapshot-9cc8813e-9d42-11e7-8bed-90b11c132b3f
  17. status:
  18.   conditions:
  19.   - lastTransitionTime: null
  20.     message: Snapshot created successfully
  21.     reason: ""
  22.     status: "True"
  23.     type: Ready
  24.   creationTimestamp: null

Restore Snapshot

To restore a PV from a VolumeSnapshot, create a PVC:

  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   name: snapshot-pv-provisioning-demo
  5.   annotations:
  6.     snapshot.alpha.kubernetes.io/snapshot: snapshot-demo
  7. spec:
  8.   storageClassName: snapshot-promoter

annotations: snapshot.alpha.kubernetes.io/snapshot is the name of the VolumeSnapshot to be restored. storageClassName: StorageClass is created by the administrator for restoring VolumeSnapshots.

A new PersistentVolume is created and bound to the PersistentVolumeClaim. The process may take several minutes depending on the PV type.

Delete Snapshot

To delete a snapshot-demo:

  1. $ oc delete volumesnapshot/snapshot-demo

The VolumeSnapshotData bound to the VolumeSnapshot is automatically deleted.