Upgrading Linkerd

In this guide, we’ll walk you through how to upgrade Linkerd.

The upgrade notices below contain important information you need to be aware ofbefore commencing with the upgrade process:

Upgrade notice: stable-2.6.0

NoteUpgrading to this release from edge-19.9.3, edge-19.9.4, edge-19.9.5 andedge-19.10.1 will incur data plane downtime, due to a recent change introducedto ensure zero downtime upgrade for previous stable releases.

The destination container is now deployed as its own Deployment workload.Once your control plane is successfully upgraded, your data plane must berestarted.

If you have previously labelled any of your namespaces with thelinkerd.io/is-control-plane label so that their pod creation events areignored by the HA proxy injector, you will need to update these namespacesto use the new config.linkerd.io/admission-webhooks: disabled label.

When ready, you can begin the upgrade process byinstalling the new CLI.

Upgrade notice: stable-2.5.0

This release supports Kubernetes 1.12+.

NoteLinkerd 2.5.0 introduced Helm support. If Linkerd wasinstalled via linkerd install, it must be upgraded via linkerd upgrade. IfLinkerd was installed via Helm, it must be upgraded via Helm. Mixing these twoinstallation procedures is not supported.

Upgrading from stable-2.4.x

NoteThese instructions also apply to upgrading from edge-19.7.4, edge-19.7.5,edge-19.8.1, edge-19.8.2, edge-19.8.3, edge-19.8.4, and edge-19.8.5.

Use the linkerd upgrade command to upgrade the control plane. This commandensures that all of the control plane’s existing configuration and mTLS secretsare retained.

  1. # get the latest stable CLI
  2. curl -sL https://run.linkerd.io/install | sh
  1. linkerd upgrade | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

The options —prune -l linkerd.io/control-plane-ns=linkerd above make surethat any resources that are removed from the linkerd upgrade output, areeffectively removed from the system.

For upgrading a multi-stage installation setup, follow the instructions atUpgrading a multi-stage install.

Users who have previously saved the Linkerd control plane’s configuration tofiles can follow the instructions atUpgrading via manifeststo ensure those configuration are retained by the linkerd upgrade command.

Once the upgrade command completes, use the linkerd check command to confirmthe control plane is ready.

Note

The stable-2.5 linkerd check command will return an error when run againstan older control plane. This error is benign and will resolve itself once thecontrol plane is upgraded to stable-2.5:

  1. linkerd-config

√ control plane Namespace exists√ control plane ClusterRoles exist√ control plane ClusterRoleBindings exist× control plane ServiceAccounts exist missing ServiceAccounts: linkerd-heartbeat see https://linkerd.io/checks/#l5d-existence-sa for hints

When ready, proceed to upgrading the data plane by following the instructions atUpgrade the data plane.

Upgrade notice: stable-2.4.0

This release supports Kubernetes 1.12+.

Upgrading from stable-2.3.x, edge-19.4.5, edge-19.5.x, edge-19.6.x, edge-19.7.x

Use the linkerd upgrade command to upgrade the control plane. This commandensures that all of the control plane’s existing configuration and mTLS secretsare retained.

  1. # get the latest stable CLI
  2. curl -sL https://run.linkerd.io/install | sh

For Kubernetes 1.12+:

  1. linkerd upgrade | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

For Kubernetes pre-1.12 where the mutating and validating webhookconfigurations’ sideEffects fields aren’t supported:

  1. linkerd upgrade --omit-webhook-side-effects | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

The sideEffects field is added to the Linkerd webhook configurations toindicate that the webhooks have no side effects on other resources.

For HA setup, the linkerd upgrade command will also retain all previous HAconfiguration. Note that the mutating and validating webhook configurations areupdated to set their failurePolicy fields to fail to ensure that un-injectedworkloads (as a result of unexpected errors) are rejected during the admissionprocess. The HA mode has also been updated to schedule multiple replicas of thelinkerd-proxy-injector and linkerd-sp-validator deployments.

For users upgrading from the edge-19.5.3 release, note that the upgradeprocess will fail with the following error message, due to a naming bug:

  1. The ClusterRoleBinding "linkerd-linkerd-tap" is invalid: roleRef: Invalid value:
  2. rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole",
  3. Name:"linkerd-linkerd-tap"}: cannot change roleRef

This can be resolved by simply deleting the linkerd-linkerd-tap cluster rolebinding resource, and re-running the linkerd upgrade command:

  1. kubectl delete clusterrole/linkerd-linkerd-tap

For upgrading a multi-stage installation setup, follow the instructions atUpgrading a multi-stage install.

Users who have previously saved the Linkerd control plane’s configuration tofiles can follow the instructions atUpgrading via manifeststo ensure those configuration are retained by the linkerd upgrade command.

Once the upgrade command completes, use the linkerd check command to confirmthe control plane is ready.

Note

The stable-2.4 linkerd check command will return an error when run againstan older control plane. This error is benign and will resolve itself once thecontrol plane is upgraded to stable-2.4:

  1. linkerd-config

√ control plane Namespace exists× control plane ClusterRoles exist missing ClusterRoles: linkerd-linkerd-controller, linkerd-linkerd-identity, linkerd-linkerd-prometheus, linkerd-linkerd-proxy-injector, linkerd-linkerd-sp-validator, linkerd-linkerd-tap see https://linkerd.io/checks/#l5d-existence-cr for hints

When ready, proceed to upgrading the data plane by following the instructions atUpgrade the data plane.

Upgrading from stable-2.2.x

Follow the stable-2.3.0 upgrade instructionsto upgrade the control plane to the stable-2.3.2 release first. Then followthese instructionsto upgrade the stable-2.3.2 control plane to stable-2.4.0.

Upgrade notice: stable-2.3.0

stable-2.3.0 introduces a new upgrade command. This command only works forthe edge-19.4.x and newer releases. When using the upgrade command fromedge-19.2.x or edge-19.3.x, all the installation flags previously providedto the install command must also be added.

Upgrading from stable-2.2.x

To upgrade from the stable-2.2.x release, follow theStep-by-step instructions.

Note that if you had previously installed Linkerd with —tls=optional, deletethe linkerd-ca deployment after successful Linkerd control plane upgrade:

  1. kubectl -n linkerd delete deploy/linkerd-ca

Upgrading from edge-19.4.x

  1. # get the latest stable
  2. curl -sL https://run.linkerd.io/install | sh
  4. # upgrade the control plane
  5. linkerd upgrade | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

Follow instructions forupgrading the data plane.

Upgrading a multi-stage install

edge-19.4.5 introduced aMulti-stage install feature. If youpreviously installed Linkerd via a multi-stage install process, you can upgradeeach stage, analogous to the original multi-stage installation process.

Stage 1, for the cluster owner:

  1. linkerd upgrade config | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

Stage 2, for the service owner:

  1. linkerd upgrade control-plane | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

Upgrading via manifests

edge-19.4.5 introduced a new —from-manifests flag to linkerd upgradeallowing manually feeding a previously saved output of linkerd install intothe command, instead of requiring a connection to the cluster to fetch theconfig:

  1. # save Linkerd installation manifest
  2. linkerd install > linkerd-install.yaml
  4. # deploy Linkerd
  5. cat linkerd-install.yaml | kubectl apply -f -
  7. # upgrade Linkerd via manifests
  8. cat linkerd-install.yaml | linkerd upgrade --from-manifests -

Alternatively, if you have already installed Linkerd without saving a manifest,you may save the relevant Linkerd resources from your existing installation foruse in upgrading later.

  1. kubectl -n linkerd get \
  2.   secret/linkerd-identity-issuer \
  3.   configmap/linkerd-config \
  4.   -oyaml > linkerd-manifests.yaml
  6. cat linkerd-manifests.yaml | linkerd upgrade --from-manifests -

Notesecret/linkerd-identity-issuer contains the trust root of Linkerd’s Identitysystem, in the form of a private key. Care should be taken if storing thisinformation on disk, such as using tools likegit-secret.

Upgrading from edge-19.2.x or edge-19.3.x

  1. # get the latest stable
  2. curl -sL https://run.linkerd.io/install | sh
  4. # Install stable control plane, using flags previously supplied during
  5. # installation.
  6. # For example, if the previous installation was:
  7. # linkerd install --proxy-log-level=warn --proxy-auto-inject | kubectl apply -f -
  8. # The upgrade command would be:
  9. linkerd upgrade --proxy-log-level=warn --proxy-auto-inject | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

Follow instructions forupgrading the data plane.

Upgrade notice: stable-2.2.0

There are two breaking changes in stable-2.2.0. One relates toService Profiles, the other relates toAutomatic Proxy Injection. If you are not usingeither of these features, you may skip directlyto the full upgrade instructions.

Service Profile namespace location

Service Profiles, previously defined in thecontrol plane namespace in stable-2.1.0, are now defined in their respectiveclient and server namespaces. Service Profiles defined in the client namespacetake priority over ones defined in the server namespace.

Automatic Proxy Injection opt-in

The linkerd.io/inject annotation, previously opt-out in stable-2.1.0, is nowopt-in.

To enable automation proxy injection for a namespace, you must enable thelinkerd.io/inject annotation on either the namespace or the pod spec. For moredetails, see the Automatic Proxy Injection doc.

A note about application updates

Also note that auto-injection only works during resource creation, not update.To update the data plane proxies of a deployment that was auto-injected, do oneof the following:

  • Manually re-inject the application via linkerd inject (more info below underUpgrade the data plane)
  • Delete and redeploy the applicationAuto-inject support for application updates is tracked ongithub

Upgrade the CLI

This will upgrade your local CLI to the latest version. You will want to followthese instructions for anywhere that uses the linkerd CLI. For Helm users feelfree to skip to the Helm section.

To upgrade the CLI locally, run:

  1. curl -sL https://run.linkerd.io/install | sh

Alternatively, you can download the CLI directly via theLinkerd releases page.

Verify the CLI is installed and running correctly with:

  1. linkerd version --client

Which should display:

  1. Client version: stable-2.6.0

NoteUntil you upgrade the control plane, some new CLI commands may not work.

You are now ready to upgrade your control plane.

Upgrade the Control Plane

Now that you have upgraded the CLI, it is time to upgrade the Linkerd controlplane on your Kubernetes cluster. Don’t worry, the existing data plane willcontinue to operate with a newer version of the control plane and your meshedservices will not go down.

NoteYou will lose the historical data from Prometheus. If you would like to havethat data persisted through an upgrade, take a look at thepersistence documentation

With Linkerd CLI

Use the linkerd upgrade command to upgrade the control plane. This commandensures that all of the control plane’s existing configuration and mTLS secretsare retained.

  1. linkerd upgrade | kubectl apply --prune -l linkerd.io/control-plane-ns=linkerd -f -

For upgrading a multi-stage installation setup, follow the instructions atUpgrading a multi-stage install.

Users who have previously saved the Linkerd control plane’s configuration tofiles can follow the instructions atUpgrading via manifeststo ensure those configuration are retained by the linkerd upgrade command.

With Helm

For a Helm workflow, check out the instructions atHelm upgrade procedure.

Verify the control plane upgrade

Once the upgrade process completes, check to make sure everything is healthyby running:

  1. linkerd check

This will run through a set of checks against your control plane and make surethat it is operating correctly.

To verify the Linkerd control plane version, run:

  1. linkerd version

Which should display:

  1. Client version: stable-2.6.0
  2. Server version: stable-2.6.0

Next, we will upgrade your data plane.

Upgrade the Data Plane

With a fully up-to-date CLI running locally and Linkerd control plane running onyour Kubernetes cluster, it is time to upgrade the data plane. This will changethe version of the linkerd-proxy sidecar container and run a rolling deploy onyour service.

Note

With kubectl 1.15+, you can use the kubectl rollout restart command torestart all your meshed services. For example,

  1. kubectl -n <namespace> rollout restart deploy

As the pods are being re-created, the proxy injector will auto-inject the newversion of the proxy into the pods.

Workloads that were previously injected using the linkerd inject —manualcommand can be upgraded by re-injecting the applications in-place. For example,

  1. kubectl -n emojivoto get deploy -l linkerd.io/control-plane-ns=linkerd -oyaml \
  2.   | linkerd inject --manual - \
  3.   | kubectl apply -f -

Verify the data plane upgrade

Check to make sure everything is healthy by running:

  1. linkerd check --proxy

This will run through a set of checks against both your control plane and dataplane to verify that it is operating correctly.

You can make sure that you’ve fully upgraded all the data plane by running:

  1. kubectl get po --all-namespaces -o yaml \
  2.   | grep linkerd.io/proxy-version

The output will look something like:

  1. linkerd.io/proxy-version: stable-2.6.0
  2. linkerd.io/proxy-version: stable-2.6.0

Congratulation! You have successfully upgraded your Linkerd to the newerversion. If you have any questions, feel free to raise them at the #linkerd2channel in the Linkerd slack.

Step-by-step instructions (stable-2.2.x)

Upgrade the 2.2.x CLI

This will upgrade your local CLI to the latest version. You will want to followthese instructions for anywhere that uses the linkerd CLI.

To upgrade the CLI locally, run:

  1. curl -sL https://run.linkerd.io/install | sh

Alternatively, you can download the CLI directly via theLinkerd releases page.

Verify the CLI is installed and running correctly with:

  1. linkerd version

Which should display:

  1. Client version: stable-2.6.0
  2. Server version: stable-2.1.0

It is expected that the Client and Server versions won’t match at this point inthe process. Nothing has been changed on the cluster, only the local CLI hasbeen updated.

NoteUntil you upgrade the control plane, some new CLI commands may not work.

Upgrade the 2.2.x control plane

Now that you have upgraded the CLI running locally, it is time to upgrade theLinkerd control plane on your Kubernetes cluster. Don’t worry, the existing dataplane will continue to operate with a newer version of the control plane andyour meshed services will not go down.

To upgrade the control plane in your environment, run the following command.This will cause a rolling deploy of the control plane components that havechanged.

  1. linkerd install | kubectl apply -f -

The output will be:

  1. namespace/linkerd configured
  2. configmap/linkerd-config created
  3. serviceaccount/linkerd-identity created
  4. clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity configured
  5. clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity configured
  6. service/linkerd-identity created
  7. secret/linkerd-identity-issuer created
  8. deployment.extensions/linkerd-identity created
  9. serviceaccount/linkerd-controller unchanged
  10. clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-controller configured
  11. clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-controller configured
  12. service/linkerd-controller-api configured
  13. service/linkerd-destination created
  14. deployment.extensions/linkerd-controller configured
  15. customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io configured
  16. serviceaccount/linkerd-web unchanged
  17. service/linkerd-web configured
  18. deployment.extensions/linkerd-web configured
  19. serviceaccount/linkerd-prometheus unchanged
  20. clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-prometheus configured
  21. clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-prometheus configured
  22. service/linkerd-prometheus configured
  23. deployment.extensions/linkerd-prometheus configured
  24. configmap/linkerd-prometheus-config configured
  25. serviceaccount/linkerd-grafana unchanged
  26. service/linkerd-grafana configured
  27. deployment.extensions/linkerd-grafana configured
  28. configmap/linkerd-grafana-config configured
  29. serviceaccount/linkerd-sp-validator created
  30. clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator configured
  31. clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-sp-validator configured
  32. service/linkerd-sp-validator created
  33. deployment.extensions/linkerd-sp-validator created

Check to make sure everything is healthy by running:

  1. linkerd check

This will run through a set of checks against your control plane and make surethat it is operating correctly.

To verify the Linkerd control plane version, run:

  1. linkerd version

Which should display:

  1. Client version: stable-2.6.0
  2. Server version: stable-2.6.0

NoteYou will lose the historical data from Prometheus. If you would like to havethat data persisted through an upgrade, take a look at thepersistence documentation

Upgrade the 2.2.x data plane

With a fully up-to-date CLI running locally and Linkerd control plane running onyour Kubernetes cluster, it is time to upgrade the data plane. This will changethe version of the linkerd-proxy sidecar container and run a rolling deploy onyour service.

For stable-2.3.0+, if your workloads are annotated with the auto-injectlinkerd.io/inject: enabled annotation, then you can just restart your podsusing your Kubernetes cluster management tools (helm, kubectl etc.).

Note

With kubectl 1.15+, you can use the kubectl rollout restart command torestart all your meshed services. For example,

  1. kubectl -n <namespace> rollout restart deploy

As the pods are being re-created, the proxy injector will auto-inject the newversion of the proxy into the pods.

If auto-injection is not part of your workflow, you can still manually upgradeyour meshed services by re-injecting your applications in-place.

Begin by retrieving your YAML resources via kubectl, and pass them through thelinkerd inject command. This will update the pod spec with thelinkerd.io/inject: enabled annotation. This annotation will be picked up byLinkerd’s proxy injector during the admission phase where the Linkerd proxy willbe injected into the workload. By using kubectl apply, Kubernetes will do arolling deploy of your service and update the running pods to the latestversion.

Example command to upgrade an application in the emojivoto namespace, composedof deployments:

  1. kubectl -n emojivoto get deploy -l linkerd.io/control-plane-ns=linkerd -oyaml \
  2.   | linkerd inject - \
  3.   | kubectl apply -f -

Check to make sure everything is healthy by running:

  1. linkerd check --proxy

This will run through a set of checks against both your control plane and dataplane to verify that it is operating correctly.

You can make sure that you’ve fully upgraded all the data plane by running:

  1. kubectl get po --all-namespaces -o yaml \
  2.   | grep linkerd.io/proxy-version

The output will look something like:

  1. linkerd.io/proxy-version: stable-2.6.0
  2. linkerd.io/proxy-version: stable-2.6.0

If there are any older versions listed, you will want to upgrade them as well.