Longhorn can automatically reattach then remount volumes if an unexpected detachment happens, which can happen during a Kubernetes upgrade or a Docker reboot.

Note: This section assumes familiarity with Linux storage concepts such as attaching and mounting volumes, and Kubernetes configuration of persistent volume storage.

To enable Longhorn to restart workloads after automatically reattaching and remounting volumes

After reattachment and remount are complete, you may need to manually restart the related workload containers for the volume restoration if the following recommended setup is not applied.

  • In a reattachment, Longhorn will reattach the volume if the volume engine dies unexpectedly.
  • In a remount, Longhorn will detect and remount the filesystem for the volume after the reattachment.

Requirements

The auto remount does not work for xfs filesystem.

Mounting one more layers with the xfs filesystem is not allowed and will trigger the error XFS (sdb): Filesystem has duplicate UUID <filesystem UUID> - can't mount.

If you use the xfs filesystem, you will need to manually unmount, then mount the xfs filesystem on the host. The device path on the host for the attached volume is /dev/longhorn/<volume name> .

Automatically Remount Volumes and Restart Workloads

In order to recover unexpectedly detached volumes automatically, set restartPolicy to Always, then add livenessProbe for the workloads using Longhorn volumes.

Then those workloads will be restarted automatically after reattachment and remount.

Here is one example for the setup:

  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   name: longhorn-volv-pvc
  5. spec:
  6.   accessModes:
  7.     - ReadWriteOnce
  8.   storageClassName: longhorn
  9.   resources:
  10.     requests:
  11.       storage: 2Gi
  12. ---
  13. apiVersion: v1
  14. kind: Pod
  15. metadata:
  16.   name: volume-test
  17.   namespace: default
  18. spec:
  19.   restartPolicy: Always
  20.   containers:
  21.   - name: volume-test
  22.     image: nginx:stable-alpine
  23.     imagePullPolicy: IfNotPresent
  24.     livenessProbe:
  25.       exec:
  26.         command:
  27.         - ls
  28.         - /data/lost+found
  29.       initialDelaySeconds: 5
  30.       periodSeconds: 5
  31.     volumeMounts:
  32.     - name: volv
  33.       mountPath: /data
  34.     ports:
  35.     - containerPort: 80
  36.   volumes:
  37.   - name: volv
  38.     persistentVolumeClaim:
  39.       claimName: longhorn-volv-pvc
  • The directory used in the livenessProbe will be <volumeMount.mountPath>/lost+found
  • Don’t set a short interval for livenessProbe.periodSeconds, e.g., 1s. The liveness command is CPU consuming.

Manually Restart Workload Containers

This solution is applied only if:

  • The Longhorn volume is reattached and remounted automatically.
  • The above setup is not included when the related workload is launched. In this case, the volume mount propagation is not Bidirectional, and the Longhorn remount operation won’t be propagated to the workload containers if the containers are not restarted.

To restart the workload containers,

  1. Figure out on which node the related workload’s containers are running
  1. kubectl -n <namespace of your workload> get pods <workload's pod name> -o wide
  1. Connect to the node. e.g., ssh
  2. Figure out the containers belonging to the workload
  1. docker ps

By checking the columns COMMAND and NAMES of the output, you can find the corresponding container

  1. Restart the container
  1. docker restart <the container ID of the workload>