Upgrade Kubeflow

Upgrading your Kubeflow installation on Google Cloud

Before you start

To better understand upgrade process, you should read the following sections first:

This guide assumes the following settings:

  • The ${MGMT_DIR} and ${MGMT_NAME} environment variables are the same as in Management cluster setup.
  • The ${KF_NAME}, ${CLIENT_ID} and ${CLIENT_SECRET} environment variables are the same as in Deploy using kubectl and kpt.
  • The ${KF_DIR} environment variable contains the path to your Kubeflow application directory, which holds your Kubeflow configuration files. For example, /opt/gcp-blueprints/kubeflow/.

General upgrade instructions

Starting from Kubeflow v1.3, we have reworked on the structure of kubeflow/gcp-blueprints repository. All resources are located in gcp-blueprints/management directory. Upgrade to Management cluster v1.3 is not supported.

Before Kubeflow v1.3, both management cluster and Kubeflow cluster follow the same instance and upstream folder convention. To upgrade, you’ll typically need to update packages in upstream to the new version and repeat the make apply-<subcommand> commands in their respective deployment process.

However, specific upgrades might need manual actions below.

Upgrading management cluster

General instruction for upgrading management cluster

It is strongly recommended to use source control to keep a copy of your working repository for recording changes at each step.

Due to the refactoring of kubeflow/manifests repository, the way we depend on kubeflow/gcp-blueprints has changed drastically. This section suits for upgrading from Kubeflow 1.3 to higher.

  1. The instructions below assume that your current working directory is

    1. cd "${MGMT_DIR}"

  2. Use your management cluster’s kubectl context:

    1. # Look at all your contexts
    2. kubectl config get-contexts
    3. # Select your management cluster's context
    4. kubectl config use-context "${MGMT_NAME}"
    5. # Verify the context connects to the cluster properly
    6. kubectl get namespace

    If you are using a different enviroment, you can always reconfigure the context by:

    1. make create-context

  3. Check your existing config connector version:

    1. # For Kubeflow v1.3, it should be 1.46.0
    2. $ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
    3. 1.46.0

  4. Merge the content from new Kubeflow version of kubeflow/gcp-blueprints

    1. WORKING_BRANCH=<your-github-working-branch>
    2. VERSION_TAG=<targeted-kubeflow-version-tag-on-github>
    3. git checkout -b "${WORKING_BRANCH}"
    4. git remote add upstream https://github.com/kubeflow/gcp-blueprints.git # This is one time only.
    5. git fetch upstream 
    6. git merge "${VERSION_TAG}"

  5. Make sure your build directory (./build by default) is checked in to source control (git).

  6. Run the following command to hydrate Config Connector resources:

    1. make hydrate-kcc

  7. Compare the difference on your source control tracking after making hydration change. If they are addition or modification only, proceed to next step. If it includes deletion, you need to use kubectl delete to manually clean up the deleted resource for cleanup.

  8. After confirmation, run the following command to apply new changes:

    1. make apply-kcc

  9. Check version has been upgraded after applying new Config Connector resource:

    1. $ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'

Upgrade management cluster from v1.1 to v1.2

  1. The instructions below assume that your current working directory is

    1. cd "${MGMT_DIR}"

  2. Use your management cluster’s kubectl context:

    1. # Look at all your contexts
    2. kubectl config get-contexts
    3. # Select your management cluster's context
    4. kubectl config use-context "${MGMT_NAME}"
    5. # Verify the context connects to the cluster properly
    6. kubectl get namespace

    If you are using a different enviroment, you can always reconfigure the context by:

    1. make create-context

  3. Check your existing config connector version:

    1. # For Kubeflow v1.1, it should be 1.15.1
    2. $ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
    3. 1.15.1

  4. Uninstall the old config connector in the management cluster:

    1. kubectl delete sts,deploy,po,svc,roles,clusterroles,clusterrolebindings --all-namespaces -l cnrm.cloud.google.com/system=true --wait=true
    2. kubectl delete validatingwebhookconfiguration abandon-on-uninstall.cnrm.cloud.google.com --ignore-not-found --wait=true
    3. kubectl delete validatingwebhookconfiguration validating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true
    4. kubectl delete mutatingwebhookconfiguration mutating-webhook.cnrm.cloud.google.com --ignore-not-found --wait=true

    These commands uninstall the config connector without removing your resources.

  5. Replace your ./Makefile with the version in Kubeflow v1.2.0: https://github.com/kubeflow/gcp-blueprints/blob/v1.2.0/management/Makefile.

    If you made any customizations in ./Makefile, you should merge your changes with the upstream version. We’ve refactored the Makefile to move substantial commands into the upstream package, so hopefully future upgrades won’t require a manual merge of the Makefile.

  6. Update ./upstream/management package:

    1. make update

  7. Use kpt to set user values:

    1. kpt cfg set -R . name ${MGMT_NAME}
    2. kpt cfg set -R . gcloud.core.project ${MGMT_PROJECT}
    3. kpt cfg set -R . location ${LOCATION}

    Note, you can find out which setters exist in a package and what there current values are by:

    1. kpt cfg list-setters .

  8. Apply upgraded config connector:

    1. make apply-kcc

    Note, you can optionally also run make apply-cluster, but it should be the same as your existing management cluster.

  9. Check that your config connector upgrade is successful:

    1. # For Kubeflow v1.2, it should be 1.29.0
    2. $ kubectl get namespace cnrm-system -ojsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com\/version}'
    3. 1.29.0

Upgrading Kubeflow cluster

DISCLAIMERS

The upgrade process depends on each Kubeflow application to handle the upgrade properly. There’s no guarantee on data completeness unless the application provides such a guarantee.

You are recommended to back up your data before an upgrade.

Upgrading Kubeflow cluster can be a disruptive process, please schedule some downtime and communicate with your users.

To upgrade from specific versions of Kubeflow, you may need to take certain manual actions — refer to specific sections in the guidelines below.

General instructions for upgrading Kubeflow cluster

  1. The instructions below assume that:

    • Your current working directory is:

      1. cd ${KF_DIR}

    • Your kubectl uses a context that connects to your Kubeflow cluster

      1. # List your existing contexts
      2. kubectl config get-contexts
      3. # Use the context that connects to your Kubeflow cluster
      4. kubectl config use-context ${KF_NAME}

  2. Merge the new version of kubeflow/gcp-blueprints (example: v1.3.0), you don’t need to do it again if you have already done so during management cluster upgrade.

    1. WORKING_BRANCH=<your-github-working-branch>
    2. VERSION_TAG=<targeted-kubeflow-version-tag-on-github>
    3. git checkout -b "${WORKING_BRANCH}"
    4. git remote add upstream https://github.com/kubeflow/gcp-blueprints.git # This is one time only.
    5. git fetch upstream 
    6. git merge "${VERSION_TAG}"

  3. Change the KUBEFLOW_MANIFESTS_VERSION in ./pull-upstream.sh with the targeted kubeflow version same as $VERSION_TAG. Run the following commands to pull new changes from upstream kubeflow/manifests.

    1. bash ./pull-upstream.sh

  4. Make sure you have checked in build folders for each component. The following command will change them so you can compare for difference.

    1. make hydrate

  5. Once you confirm the changes are ready to apply, run the following command to upgrade Kubeflow cluster:

    1. make apply

Note

Kubeflow on Google Cloud doesn’t guarantee the upgrade for each Kubeflow component always works with the general upgrade guide here. Please refer to corresponding repository in Kubeflow org for upgrade support.

Upgrade Kubeflow cluster to v1.3

Due to the refactoring of kubeflow/manifests repository, the way we depend on kubeflow/gcp-blueprints has changed drastically. Upgrade to Kubeflow cluster v1.3 is not supported. And individual component upgrade has been deferred to its corresponding repository for support.

Upgrade Kubeflow cluster from v1.1 to v1.2

  1. The instructions below assume

    • Your current working directory is:

      1. cd ${KF_DIR}

    • Your kubectl uses a context that connects to your Kubeflow cluster:

      1. # List your existing contexts
      2. kubectl config get-contexts
      3. # Use the context that connects to your Kubeflow cluster
      4. kubectl config use-context ${KF_NAME}

  2. (Recommended) Replace your ./Makefile with the version in Kubeflow v1.2.0: https://github.com/kubeflow/gcp-blueprints/blob/v1.2.0/kubeflow/Makefile.

    If you made any customizations in ./Makefile, you should merge your changes with the upstream version.

    This step is recommended, because we introduced usability improvements and fixed compatibility for newer Kustomize versions (while still being compatible with Kustomize v3.2.1) to the Makefile. However, the deployment process is backward-compatible, so this is recommended, but not required.

  3. Update ./upstream/manifests package:

    1. make update

  4. Before applying new resources, you need to delete some immutable resources that were updated in this release:

    1. kubectl delete statefulset kfserving-controller-manager -n kubeflow --wait
    2. kubectl delete crds experiments.kubeflow.org suggestions.kubeflow.org trials.kubeflow.org

    WARNING: This step deletes all Katib running resources.

    Refer to a github comment in the v1.2 release issue for more details.

  5. Redeploy:

    1. make apply

    To evaluate the changes before deploying them you can:

    1. Run make hydrate.
    2. Compare the contents of .build with a historic version with tools like git diff.

Upgrade ASM (Anthos Service Mesh)

If you want to upgrade ASM instead of the Kubeflow components, refer to kubeflow/common/asm/Makefile for the latest instruction on upgrading ASM. Detailed explaination is also listed below. Note: If you are going to upgrade minor version or major version of ASM, it is best to read official ASM upgrade documentation first, before performing the steps below. Patch version upgrade can refer to steps below directly.

Install new ASM workload

In order to use the new ASM version, we need to download new ASM configuration package and install_asm script. Identify the target ASM package and script by listing the stable versions of such combination:

  1. curl https://storage.googleapis.com/csm-artifacts/asm/STABLE_VERSIONS

You will see output similar to

  1. ...
  2. 1.9.3-asm.2+config2:install_asm_1.9.3-asm.2-config2
  3. 1.9.3-asm.2+config2:asm_vm_1.9.3-asm.2-config2
  4. 1.9.3-asm.2+config1:install_asm_1.9.3-asm.2-config1
  5. 1.9.3-asm.2+config1:asm_vm_1.9.3-asm.2-config1
  6. 1.9.2-asm.1+config4:install_asm_1.9.2-asm.1-config4
  7. 1.9.2-asm.1+config4:asm_vm_1.9.2-asm.1-config4
  8. ...

Taking the first line as example, you can split this line by :, use the first half for ASM configuration package version, and the second half for install_asm script version. Therefore, assigning these two values to ${ASM_PACKAGE_VERSION} and ${INSTALL_ASM_SCRIPT_VERSION} in kubeflow/common/asm/Makefile like below:

  1. ASM_PACKAGE_VERSION=1.9.3-asm.2+config2
  2. INSTALL_ASM_SCRIPT_VERSION=install_asm_1.9.3-asm.2-config2

Then run following command under path kubeflow/common/asm to install new ASM, the old ASM will not be uninstalled:

  1. make apply

Once installed successfully, you can see istiod Deployment in your cluster with name in pattern istiod-asm-VERSION-REVISION, for example: istiod-asm-193-2.

Upgrade for other Kubeflow components

There are multiple Kubeflow components with ASM namespace label, including user created namespaces. To upgrade them at once, change the following line in kubeflow/env.sh with targeted ASM version asm-VERSION-REVISION, like asm-193-2.

  1. export ASM_LABEL=asm-193-2

Then run following commands under kubeflow/ path to configure environment variables:

  1. source env.sh

Run the following command to configure kpt setter:

  1. bash kpt-set.sh

Examine the change using source control after running the following command:

  1. make hydrate

Refer to Deploying and redeploying workloads for the complete steps to adopt new ASM version. As part of the above instruction, you can run following command to update namespaces’ labels across the cluster:

  1. make apply

(Optional) Uninstall old ASM workload

Once you validated that new ASM installation and sidecar-injection for Kubeflow components are working as expected. You can follow the Complete the transition at Deploying and redeploying workloads to remove the old ASM deployments.

Last modified 17.05.2021: (gke) Full Kubeflow Deployment guide on GKE (revision for Kubeflow 1.3) (#2686) (f50693a3)