Upgrade from v1.1.2 to v1.2.0

General information

Upgrade from v1.1.2 to v1.2.0 - 图1tip

Before you start an upgrade, you can run the pre-check script to make sure the cluster is in a stable state. For more details, please visit this URL for the script.

Once there is an upgradable version, the Harvester GUI Dashboard page will show an upgrade button. For more details, please refer to start an upgrade.

For the air-gap env upgrade, please refer to prepare an air-gapped upgrade.

Known issues

1. An upgrade can’t start and reports "validator.harvesterhci.io" denied the request: managed chart rancher-monitoring is not ready, please wait for it to be ready

If a cluster is configured with a storage network, an upgrade can’t start with the following message.

Upgrade from v1.1.2 to v1.2.0 - 图2

2. An upgrade is stuck in Creating Upgrade Repository

During an upgrade, Creating Upgrade Repository is stuck in the Pending state:

Upgrade from v1.1.2 to v1.2.0 - 图3

Please perform the following steps to check if the cluster runs into the issue:

  1. Check the upgrade repository pod:

    Upgrade from v1.1.2 to v1.2.0 - 图4

    If the virt-launcher-upgrade-repo-hvst-<upgrade-name> pod stays in ContainerCreating, your cluster might have run into this issue. In this case, proceed with step 2.

  2. Check the upgrade repository volume in the Longhorn GUI.

    1. Go to Longhorn GUI.

    2. Navigate to the Volume page.

    3. Check the upgrade repository VM volume. It should be attached to a pod called virt-launcher-upgrade-repo-hvst-<upgrade-name>. If one of the volume’s replicas stays in Stopped (gray color), the cluster is running into the issue.

      Upgrade from v1.1.2 to v1.2.0 - 图5

3. An upgrade is stuck when pre-draining a node

Starting from v1.1.0, Harvester will wait for all volumes to become healthy (when node count >= 3) before upgrading a node. Generally, you can check volumes’ health if an upgrade is stuck in the “pre-draining” state.

Visit “Access Embedded Longhorn” to see how to access the embedded Longhorn GUI.

You can also check the pre-drain job logs. Please refer to Phase 4: Upgrade nodes in the troubleshooting guide.

4. An upgrade is stuck in upgrading the first node: Job was active longer than the specified deadline

An upgrade fails, as shown in the screenshot below:

Upgrade from v1.1.2 to v1.2.0 - 图6

5. An upgrade is stuck in the Pre-drained state

You might see an upgrade is stuck in the “pre-drained” state:

Upgrade from v1.1.2 to v1.2.0 - 图7

In this stage, Kubernetes is supposed to drain the workload on the node, but some reasons might cause the process to stall.

5.1 The node contains a Longhorn instance-manager-r pod that serves single-replica volume(s)

Longhorn doesn’t allow draining a node if the node contains the last surviving replica of a volume. To check if a node is running into this situation, follow these steps:

  1. List single-replica volumes with the command:

    1. kubectl get volumes.longhorn.io -A -o yaml | yq '.items[] | select(.spec.numberOfReplicas == 1) | .metadata.namespace + "/" + .metadata.name'

    For example:

    1. $ kubectl get volumes.longhorn.io -A -o yaml | yq '.items[] | select(.spec.numberOfReplicas == 1) | .metadata.namespace + "/" + .metadata.name'
    2. longhorn-system/pvc-d1f19bab-200e-483b-b348-c87cfbba85ab

  2. Check if the replica resides on the stuck node:

    List the NodeID of the volume’s replica with the command:

    1. kubectl get replicas.longhorn.io -n longhorn-system -o yaml | yq '.items[] | select(.metadata.labels.longhornvolume == "<volume>") | .spec.nodeID'

    For example:

    1. $ kubectl get replicas.longhorn.io -n longhorn-system -o yaml | yq '.items[] | select(.metadata.labels.longhornvolume == "pvc-d1f19bab-200e-483b-b348-c87cfbba85ab") | .spec.nodeID'
    2. node1

    If the result shows that the replica resides on the node where the upgrade is stuck (in this example, node1), your cluster is hitting this issue.

There are a couple of ways to address this situation. Choose the most appropriate method for your VM:

  1. Shut down the VM that uses the single-replica volume to detach the volume, allowing the upgrade to continue.
  2. Adjust the volumes’s replicas to more than one.
    1. Go to Longhorn GUI.
    2. Go to the Volume page.
    3. Locate the problematic volume and click the icon on the right side, then select Update Replicas Count: Upgrade from v1.1.2 to v1.2.0 - 图8
    4. Increase the Number of Replicas and select OK.

5.2 Misconfigured Longhorn instance-manager-r Pod Disruption Budgets (PDB)

A misconfigured PDB could cause this issue. To check if that’s the case, perform the following steps:

  1. Assume the stuck node is harvester-node-1.

  2. Check the instance-manager-e or instance-manager-r pod names on the stuck node:

    1. $ kubectl get pods -n longhorn-system --field-selector spec.nodeName=harvester-node-1 | grep instance-manager
    2. instance-manager-r-d4ed2788          1/1     Running   0              3d8h

    The output above shows that the instance-manager-r-d4ed2788 pod is on the node.

  3. Check Rancher logs and verify that the instance-manager-e or instance-manager-r pod can’t be drained:

    1. $ kubectl logs deployment/rancher -n cattle-system
    2. ...
    3. 2023-03-28T17:10:52.199575910Z 2023/03/28 17:10:52 [INFO] [planner] rkecluster fleet-local/local: waiting: draining etcd node(s) custom-4f8cb698b24a,custom-a0f714579def
    4. 2023-03-28T17:10:55.034453029Z evicting pod longhorn-system/instance-manager-r-d4ed2788
    5. 2023-03-28T17:10:55.080933607Z error when evicting pods/"instance-manager-r-d4ed2788" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

  4. Run the command to check if there is a PDB associated with the stuck node:

    1. $ kubectl get pdb -n longhorn-system -o yaml | yq '.items[] | select(.spec.selector.matchLabels."longhorn.io/node"=="harvester-node-1") | .metadata.name'
    2. instance-manager-r-466e3c7f

  5. Check the owner of the instance manager to this PDB:

    1. $ kubectl get instancemanager instance-manager-r-466e3c7f -n longhorn-system -o yaml | yq -e '.spec.nodeID'
    2. harvester-node-2

    If the output doesn’t match the stuck node (in this example output, harvester-node-2 doesn’t match the stuck node harvester-node-1), then we can conclude this issue happens.

  6. Before applying the workaround, check if all volumes are healthy:

    1. kubectl get volumes -n longhorn-system -o yaml | yq '.items[] | select(.status.state == "attached")| .status.robustness'

    The output should all be healthy. If this is not the case, you might want to uncordon nodes to make the volume healthy again.

  7. Remove the misconfigured PDB:

    1. kubectl delete pdb instance-manager-r-466e3c7f -n longhorn-system

5.3 The instance-manager-e pod could not be drained

During an upgrade, you might encounter an issue where you can’t drain the instance-manager-e pod. When this situation occurs, you will see error messages in the Rancher logs like the ones shown below:

  1. $ kubectl logs deployment/rancher -n cattle-system | grep "evicting pod"
  2. evicting pod longhorn-system/instance-manager-r-a06a43f3437ab4f643eea7053b915a80
  3. evicting pod longhorn-system/instance-manager-e-452e87d2
  4. error when evicting pods/"instance-manager-r-a06a43f3437ab4f643eea7053b915a80" -n "Longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
  5. error when evicting pods/"instance-manager-e-452e87d2" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Check the instance-manager-e to see if any engine instances remain.

  1. $ kubectl get instancemanager instance-manager-e-452e87d2 -n longhorn-system -o yaml | yq -e ".status.instances"
  2. pvc-7b120d60-1577-4716-be5a-62348271025a-e-1cd53c57:
  3.   spec:
  4.     name: pvc-7b120d60-1577-4716-be5a-62348271025a-e-1cd53c57
  5.   status:
  6.     endpoint: ""
  7.     errorMsg: ""
  8.     listen: ""
  9.     portEnd: 10001
  10.     portStart: 10001
  11.     resourceVersion: 0
  12.     state: running
  13.     type: ""

In this example, the instance-manager-e-452e87d2 still has an engine instance, so you can’t drain the pod.

You need to check the engine numbers to see if any engine number is redundant. Each PVC should only have one engine.

  1. # kubectl get engines -n longhorn-system -l longhornvolume=pvc-7b120d60-1577-4716-be5a-62348271025a
  2. NAME                                                  STATE     NODE               INSTANCEMANAGER                                      IMAGE                               AGE
  3. pvc-76120d60-1577-4716-be5a-62348271025a-e-08220662   running   harvester-qv4hd    instance-manager-e-625d715e2f2e7065d64339f9b31407c2  longhornio/longhorn-engine:v1.4.3   2d12h
  4. pvc-7b120d60-1577-4716-be5a-62348271025a-e-lcd53c57   running   harvester-lhlkv    instance-manager-e-452e87d2                          longhornio/longhorn-engine:v1.4.3   4d10h

The example above shows that two engines exist for the same PVC, which is a known issue in Longhorn #6642. To resolve this, delete the redundant engine to allow the upgrade to continue.

To determine which engine is the correct one, use the following command:

  1. $ kubectl get volumes pvc-7b120d60-1577-4716-be5a-62348271025a -n longhorn-system
  2. NAME                                      STATE     ROBUSTNESS  SCHEDULED SIZE        NODE            AGE
  3. pvc-7b120d60-1577-4716-be5a-62348271025a  attached  healthy               42949672960 harvester-q4vhd 4d10h

In this example, the volume pvc-7b120d60-1577-4716-be5a-62348271025a is active on the node harvester-q4vhd, indicating that the engine not running on this node is redundant.

To make the engine inactive and trigger its automatic deletion by Longhorn, run the following command:

  1. $ kubectl patch engine pvc-7b120d60-1577-4716-be5a-62348271025a-e-lcd53c57 -n longhorn-system --type='json' -p='[{"op": "replace", "path": "/spec/active", "value": false}]'
  2. engine.longhorn.io/pvc-7b120d60-1577-4716-be5a-62348271025a-e-lcd53c57 patched

After a few seconds, you can verify the engine’s status:

  1. $ kubectl get engine -n longhorn-system|grep pvc-7b120d60-1577-4716-be5a-62348271025a
  2. pvc-7b120d60-1577-4716-be5a-62348271025a-e-08220b62   running  harvester-q4vhd   instance-manager-e-625d715e2f2e7065d64339f9631407c2  longhornio/longhorn-engine:v1.4.3   2d13h

The instance-manager-e pod should now drain successfully, allowing the upgrade to proceed.

6. An upgrade is stuck in the Upgrading System Service state

If you notice the upgrade is stuck in the Upgrading System Service state for a long period of time, you might need to investigate if the upgrade is stuck in the apply-manifests phase.

Upgrade from v1.1.2 to v1.2.0 - 图9

  1. Check the log of the apply-manifests pod to see if the following messages repeat.

    1. $ kubectl -n harvester-system logs hvst-upgrade-md6wr-apply-manifests-wqslg --tail=10
    2. Tue Sep  5 10:20:39 UTC 2023
    3. there are still 1 pods in cattle-monitoring-system to be deleted
    4. Tue Sep  5 10:20:45 UTC 2023
    5. there are still 1 pods in cattle-monitoring-system to be deleted
    6. Tue Sep  5 10:20:50 UTC 2023
    7. there are still 1 pods in cattle-monitoring-system to be deleted
    8. Tue Sep  5 10:20:55 UTC 2023
    9. there are still 1 pods in cattle-monitoring-system to be deleted
    10. Tue Sep  5 10:21:00 UTC 2023
    11. there are still 1 pods in cattle-monitoring-system to be deleted

  2. Check if the prometheus-rancher-monitoring-prometheus-0 pod is stuck with the status Terminating.

    1. $ kubectl -n cattle-monitoring-system get pods                                  
    2. NAME                                         READY   STATUS        RESTARTS   AGE
    3. prometheus-rancher-monitoring-prometheus-0   0/3     Terminating   0          19d

  3. Find the UID of the terminating pod with the following command:

    1. $ kubectl -n cattle-monitoring-system get pod prometheus-rancher-monitoring-prometheus-0 -o jsonpath='{.metadata.uid}'
    2. 33f43165-6faa-4648-927d-69097901471c

  4. Get access to any node of the cluster via the console or SSH.

  5. Search for the related log messages in /var/lib/rancher/rke2/agent/logs/kubelet.log using the pod’s UID.

    1. E0905 10:26:18.769199   17399 reconciler.go:208] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\" (UniqueName: \"kubernetes.io/csi/driver.longhorn.io^pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\") pod \"33f43165-6faa-4648-927d-69097901471c\" (UID: \"33f43165-6faa-4648-927d-69097901471c\") : UnmountVolume.NewUnmounter failed for volume \"pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\" (UniqueName: \"kubernetes.io/csi/driver.longhorn.io^pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\") pod \"33f43165-6faa-4648-927d-69097901471c\" (UID: \"33f43165-6faa-4648-927d-69097901471c\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/vol_data.json]: open /var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume \"pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\" (UniqueName: \"kubernetes.io/csi/driver.longhorn.io^pvc-7781c988-c35b-4cf8-89e6-f2907ef33603\") pod \"33f43165-6faa-4648-927d-69097901471c\" (UID: \"33f43165-6faa-4648-927d-69097901471c\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/vol_data.json]: open /var/lib/kubelet/pods/33f43165-6faa-4648-927d-69097901471c/volumes/kubernetes.io~csi/pvc-7781c988-c35b-4cf8-89e6-f2907ef33603/vol_data.json: no such file or directory"

    If kubelet continues to complain about the volume failing to unmount, apply the following workaround to allow the upgrade to proceed.

  6. Forcibly remove the pod stuck with the status Terminating with the following command:

    1. kubectl delete pod prometheus-rancher-monitoring-prometheus-0 -n cattle-monitoring-system  --force

7. Upgrade stuck in the Upgrading System Service state

If an upgrade is stuck in an Upgrading System Service state for an extended period, some system services’ certificates may have expired. To investigate and resolve this issue, follow these steps:

  1. Find the apply-manifest job’s name with the command:

    1. kubectl get jobs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest

    Example output:

    1. NAME                                 COMPLETIONS   DURATION   AGE
    2. hvst-upgrade-9gmg2-apply-manifests   0/1           46s        46s

  2. Check the job’s log with the command:

    1. kubectl logs jobs/hvst-upgrade-9gmg2-apply-manifests -n harvester-system

    If the following messages appear in the log, continue to the next step:

    1. Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
    2. Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
    3. Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
    4. Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...

  3. Check CAPI cluster’s state with the command:

    1. kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml

    If you see a condition similar to the one below, it’s likely that the cluster has encountered the issue:

    1.     - lastUpdateTime: "2023-01-17T16:26:48Z"
    2.       message: 'configuring bootstrap node(s) custom-24cb32ce8387: waiting for probes:
    3.         kube-controller-manager, kube-scheduler'
    4.       reason: Waiting
    5.       status: Unknown
    6.       type: Updated

  4. Find the machine’s hostname with the following command, and follow the workaround to see if service certificates expire on a node:

    1. kubectl get machines.cluster.x-k8s.io -n fleet-local <machine_name> -o yaml | yq .status.nodeRef.name

    Replace <machine_name> with the machine’s name from the output in the previous step.

    Upgrade from v1.1.2 to v1.2.0 - 图10note

    If multiple nodes joined the cluster around the same time, you should perform the workaround on all those nodes.