Rotating your identity certificates

By default, the issuer certificate and trust root that Linkerd uses are validfor 365 days. If either of these certificates expires, Linkerd will no longerbe able to proxy traffic. Therefore, it is critical that you replace thesecertificates with new ones before they expire - a process called certificaterotation. This guide will show you how to achieve that without any downtime.

If your control plane is installed with thelinkerd install —identity-external-issuer command where your trust root ismanaged by a 3rd party certificate management solution like cert-manager,then this information doesn't apply to you, because it is the responsibilityof your certificate manager to rotate the certificates before they expire.

NoteAlthough Linkerd can auto-generate the trust root during installation, werecommend using your own trust root for all serious workloads, so that youhave full control over it. See Generating your own mTLS root certificateson how to do this.

Prerequisites

We are going to use step 0.13.3 andjq 1.6. We will also make use of thelinkerd check CLI utility which will give us extra assurance that the stateof our mTLS configuration is valid at any point. The minimum CLI version isedge-19.12.3.To begin with, you can run:

linkerd check --proxy

If your configuration is valid and your certificates are not expiring soon,you should see output similar to:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm√ issuer cert is within its validity period√ issuer cert is valid for at least 60 days√ issuer cert is issued by the trust root

√ data plane proxies certificate match CA

However, if you see a message warning you that your root or issuercertificates are expiring soon it means that you must perform certificaterotation before your certificates expire. If your issuer certificate is expiredyou will see an error such as:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm× issuer cert is within its validity period issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints

If your trust root has expired, you will observe the following error:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm× trust roots are within their validity period Invalid roots:

    * 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid  for hints

If encounter any of these errors, it means your cluster's TLS configurationis in invalid state. In order to address this problem skip this guide andgo straight toReplacing Expired Certificates.

Generate new trust root and issuer certificates

If you have installed Linkerd with a manually supplied trust root that is notexpiring and you have its key, then you can simply use it to generate a newissuer certificate. If this is the case, skip this step and go directly toUpdating the identity issuer certificate

First generate the root certificate with its private key. The private key needsto be stored in a secured vault so it can be used to generate a new trust rootin the future.

step certificate create identity.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure

NoteNote we use —no-password —insecure for both the roots and issuercertificates to avoid encrypting those files with a passphrase.

Bundling your original trust root with the new one

As a next step we need to bundle the trust root currently used by Linkerdtogether with the new root and configure Linkerd to use the bundle. Thefollowing command uses kubectl to fetch the Linkerd config as a jsonobject, extracts the current roots from the config with the help of jq andfinally uses step to combine it with the newly generated roots and save theresult in the bundle.crt file.

kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.global}' |  \
jq -r .identityContext.trustAnchorsPem > original-trust.crt
step certificate bundle ca-new.crt original-trust.crt bundle.crt
rm original-trust.crt

Deploying the new bundle to Linkerd

You can use linkerd upgrade in order to upgrade Linkerd to work with the newroot bundle:

linkerd upgrade  --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -

This will restart the proxies of the Linkerd components and they will bereconfigured with the new root certs. Now it is time restart the proxy for allinjected workloads in your cluster. Doing that for the emojivoto namespace forexample would look like:

kubectl -n emojivoto rollout restart deploy

Now you can run the check command to ensure that everything is ok:

linkerd check --proxy

You might have to wait a few moments until all the pods have been restarted andare configured with the correct roots. Meanwhile you might observe warnings:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm√ issuer cert is within its validity period‼ issuer cert is valid for at least 60 days issuer certificate will expire on 2019-12-19T09:51:19Z see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints√ issuer cert is issued by the trust root

    * emojivoto/emoji-d8d7d9c6b-8qwfx
    * emojivoto/vote-bot-588499c9f6-zpwz6
    * emojivoto/voting-8599548fdc-6v64k
    * emojivoto/web-67c7599f6d-xx98n
    * linkerd/linkerd-sp-validator-75f9d96dc-rch4x
    * linkerd/linkerd-tap-68d8bbf64-mpzgb
    * linkerd/linkerd-web-849f74b7c6-qlhwc
see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

When the rollout completes your check command should stop warning you thatpods need to be restarted. It will still warn you however that your issuercertificate is about to expire soon:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm√ issuer cert is within its validity period‼ issuer cert is valid for at least 60 days issuer certificate will expire on 2019-12-19T09:51:19Z see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints√ issuer cert is issued by the trust root

√ data plane proxies certificate match CA

Updating the identity issuer certificate

Now, using your trust root, generate the intermediate certificate and key pair:

step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key --ca ca-new.crt --ca-key ca-new.key --profile intermediate-ca --not-after 8760h --no-password --insecure

Provided that all proxies are updated to include a working trust root, it issafe to rotate the identity issuer certificate by using the upgrade commandagain:

linkerd upgrade  --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key | kubectl apply -f -

At this point the identity service should detect the change of the secret andautomatically update its issuer certificates. To ensure this has happened, youcan check for the specific Kubernetes event.

kubectl get events --field-selector reason=IssuerUpdated -n linkerd
LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
9s          Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

Run the check command to make sure that everything is going as expected:

linkerd check --proxy

You should see output without any certificate expiration warnings:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm√ issuer cert is within its validity period√ issuer cert is valid for at least 60 days√ issuer cert is issued by the trust root

√ data plane proxies certificate match CA

Removing the old trust root

At this point, the cert rotation process is complete. For security purposes,we will remove the old trust root from the trust bundle we created earlier.The upgrade command can do that for the Linkerd components:

linkerd upgrade  --identity-trust-anchors-file=./ca-new.crt  | kubectl apply -f -

Note that the ./ca-new.crt file is the same trust root you created at the startof this process.Additionally you can use the rollout restart command to bringthe configuration of your other injected resources up to date:

kubectl -n emojivoto rollout restart deploy
linkerd check --proxy

Finally the output of the check command should not produce any warnings orerrors:

linkerd-identity

√ certificate config is valid√ trust roots are using supported crypto algorithm√ trust roots are within their validity period√ trust roots are valid for at least 60 days√ issuer cert is using supported crypto algorithm√ issuer cert is within its validity period√ issuer cert is valid for at least 60 days√ issuer cert is issued by the trust root

√ data plane proxies certificate match CA