Installation Options for Kubeflow Pipelines

Overview of the ways to deploy Kubeflow Pipelines

Kubeflow Pipelines offers a few installation options. This page describes the options and the features available with each option:

Choosing an installation option

  1. Do you want to use other Kubeflow components in addition to Pipelines?

    If yes, choose the full Kubeflow deployment.

  2. Can you use a cloud/on-prem Kubernetes cluster?

    If you can’t, you should try using Kubeflow Pipelines on a local Kubernetes cluster for learning and testing purposes by following the steps in Deploying Kubeflow Pipelines on a local cluster.

  3. Do you want to use Kubeflow Pipelines with multi-user support?

    If yes, choose the full Kubeflow deployment with version >= v1.1.

  4. Do you deploy on Google Cloud?

    If yes, deploy Kubeflow Pipelines Standalone. You can also use Google Cloud AI Platform Pipelines to deploy Kubeflow Pipelines using a user interface, but there are limitations in customizability and upgradability. For details, please read corresponding sections.

  5. You deploy on other platforms.

    Please compare your platform specific full Kubeflow with the Kubeflow Pipelines Standalone before making your decision.

Warning: Choose your installation option with caution, there’s no current supported path to migrate data between different installation options. Please create a GitHub issue if that’s important for you.

Kubeflow Pipelines Standalone

Use this option to deploy Kubeflow Pipelines to an on-premises, cloud or even local Kubernetes cluster, without the other components of Kubeflow. To deploy Kubeflow Pipelines Standalone, you use kustomize manifests only. This process makes it simpler to customize your deployment and to integrate Kubeflow Pipelines into an existing Kubernetes cluster.

Installation guide

Kubeflow Pipelines Standalone deployment guide

Interfaces :

  • Kubeflow Pipelines UI
  • Kubeflow Pipelines SDK
  • Kubeflow Pipelines API
  • Kubeflow Pipelines endpoint is only auto-configured for Google Cloud.

If you wish to deploy Kubeflow Pipelines on other platforms, you can either access it through kubectl port-forward or configure your own platform specific auth-enabled endpoint by yourself.

Release Schedule

Kubeflow Pipelines Standalone is available for every Kubeflow Pipelines release. You will have access to the latest features.

Upgrade Support (Beta)

Upgrading Kubeflow Pipelines Standalone introduces how to upgrade in place.

Google Cloud Integrations :

Notes on specific features :

  • After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you must use a local notebook or a hosted notebook in a cloud service such as the AI Platform Notebooks.
  • Kubeflow Pipelines multi-user support is not available in standalone, because multi-user support depends on other Kubeflow components.

Full Kubeflow deployment

Use this option to deploy Kubeflow Pipelines to your local machine, on-premises, or to a cloud, as part of a full Kubeflow installation.

Installation guide

Kubeflow installation guide

Interfaces :

  • Kubeflow UI
  • Kubeflow Pipelines UI within or outside the Kubeflow UI
  • Kubeflow Pipelines SDK
  • Kubeflow Pipelines API
  • Other Kubeflow APIs
  • Kubeflow Pipelines endpoint is auto-configured with auth support for each platform

Release Schedule

The full Kubeflow is released quarterly. It has significant delay in receiving Kubeflow Pipelines updates.

Kubeflow VersionKubeflow Pipelines Version
0.7.00.1.31
1.0.00.2.0
1.0.20.2.5
1.1.01.0.0
1.2.01.0.4

Note: Google Cloud, AWS, and IBM Cloud have supported Kubeflow Pipelines 1.0.0 with multi-user separation. Other platforms might not be up-to-date for now, refer to this GitHub issue for status.

Upgrade Support : Refer to the full Kubeflow section of upgrading Kubeflow Pipelines on Google Cloud guide.

Google Cloud Integrations :

Notes on specific features :

  • After deployment, your Kubernetes cluster includes all the Kubeflow components. For example, you can use the Jupyter notebook services deployed with Kubeflow to create one or more notebook servers in your Kubeflow cluster.
  • Kubeflow Pipelines multi-user support is only available in full Kubeflow. It supports using a single Kubeflow Pipelines control plane to orchestrate user pipeline runs in multiple user namespaces with authorization.
  • Latest features and bug fixes may not be available soon because release cadence is long.

Google Cloud AI Platform Pipelines

Beta release

Google Cloud AI Platform Pipelines is currently in Beta with limited support. The Kubeflow Pipelines team is interested in any feedback you may have, in particular on the usability of the feature.

You can raise any issues or discussion items in the Kubeflow Pipelines issue tracker.

Use this option to deploy Kubeflow Pipelines to Google Kubernetes Engine (GKE) from Google Cloud Marketplace. You can deploy Kubeflow Pipelines to an existing or new GKE cluster and manage your cluster within Google Cloud.

Installation guide

Google Cloud AI Platform Pipelines documentation

Interfaces :

  • Google Cloud Console for managing the Kubeflow Pipelines cluster and other Google Cloud services
  • Kubeflow Pipelines UI via the Open Pipelines Dashboard link in the Google Cloud Console
  • Kubeflow Pipelines SDK in Cloud Notebooks
  • Kubeflow Pipelines endpoint of your instance is auto-configured for you

Release Schedule

AI Platform Pipelines is available for a chosen set of stable Kubeflow Pipelines releases. You will receive updates slightly slower than Kubeflow Pipelines Standalone.

Upgrade Support (Alpha)

An in-place upgrade is not supported.

To upgrade AI Platform Pipelines by reinstalling it (with existing data), refer to the Upgrading AI Platform Pipelines guide.

Google Cloud Integrations :

Notes on specific features :

  • After deployment, your Kubernetes cluster contains Kubeflow Pipelines only. It does not include the other Kubeflow components. For example, to use a Jupyter Notebook, you can use AI Platform Notebooks.
  • Kubeflow Pipelines multi-user support is not available in AI Platform Pipelines, because multi-user support depends on other Kubeflow components.

Last modified 03.03.2021: Move Kubeflow Pipelines under /components (#2505) (c34470b8)