Upgrade Kubernetes on Azure AKS

    AKS provides az aks upgrade for in-places nodes upgrade by node reimaged, but this will cause the original Longhorn disks missing, then there will be no disks allowing replica rebuilding in upgraded nodes anymore.

    We suggest using node-pool replacement to upgrade the agent nodes but use az aks upgrade for control plane nodes to ensure data safety.

    1. In Longhorn, set replica-replenishment-wait-interval to 0.

    2. Upgrade AKS control plane.

      1. AKS_RESOURCE_GROUP=<aks-resource-group>
      2. AKS_CLUSTER_NAME=<aks-cluster-name>
      3. AKS_K8S_VERSION_UPGRADE=<aks-k8s-version>
      5. az aks upgrade \
      6.     --resource-group ${AKS_RESOURCE_GROUP} \
      7.     --name ${AKS_CLUSTER_NAME} \
      8.     --kubernetes-version ${AKS_K8S_VERSION_UPGRADE} \
      9.     --control-plane-only

    3. Add a new node-pool.

      1. AKS_NODEPOOL_NAME_NEW=<new-nodepool-name>
      2. AKS_DISK_SIZE=<disk-size-in-gb>
      3. AKS_NODE_NUM=<number-of-nodes>
      5. az aks nodepool add \
      6.   --resource-group ${AKS_RESOURCE_GROUP} \
      7.   --cluster-name ${AKS_CLUSTER_NAME} \
      8.   --name ${AKS_NODEPOOL_NAME_NEW} \
      9.   --node-count ${AKS_NODE_NUM} \
      10.   --node-osdisk-size ${AKS_DISK_SIZE} \
      11.   --kubernetes-version ${AKS_K8S_VERSION_UPGRADE} \
      12.   --mode System

    4. Using Longhorn UI to disable the disk scheduling and request eviction for nodes in the old node-pool.

    5. Cordon and drain Kubernetes nodes in the old node-pool.

      1. AKS_NODEPOOL_NAME_OLD=<old-nodepool-name>
      3. for n in `kubectl get nodes | grep ${AKS_NODEPOOL_NAME_OLD}- | awk '{print $1}'`; do
      4.   kubectl cordon $n && \
      5.   kubectl drain $n --ignore-daemonsets --delete-emptydir-data
      6. done

    6. Delete old node-pool.

      1. az aks nodepool delete \
      2.   --cluster-name ${AKS_CLUSTER_NAME} \
      3.   --name ${AKS_NODEPOOL_NAME_OLD} \
      4.   --resource-group ${AKS_RESOURCE_GROUP}