Automatically Rotating Webhook TLS Credentials

The Linkerd control plane contains several components, called webhooks, which are called directly by Kubernetes itself. The traffic from Kubernetes to the Linkerd webhooks is secured with TLS and therefore each of the webhooks requires a secret containing TLS credentials. These certificates are different from the ones that the Linkerd proxies use to secure pod-to-pod communication and use a completely separate trust chain. For more information on rotating the TLS credentials used by the Linkerd proxies, see Automatically Rotating Control Plane TLS Credentials.

By default, when Linkerd is installed with the Linkerd CLI or with the Linkerd Helm chart, TLS credentials are automatically generated for all of the webhooks. If these certificates expire or need to be regenerated for any reason, performing a Linkerd upgrade (using the Linkerd CLI or using Helm) will regenerate them.

This workflow is suitable for most users. However, if you need these webhook certificates to be rotated automatically on a regular basis, it is possible to use cert-manager to automatically manage them.

Install Cert manager

As a first step, install cert-manager on your cluster and create the namespace that cert-manager will use to store its webhook-related resources. For simplicity, we suggest the default Linkerd control plane namespace:

  1. kubectl create namespace linkerd

Save the signing key pair as a Secret

Next, we will use the step tool, to create a signing key pair which will be used to sign each of the webhook certificates:

  1. step certificate create webhook.linkerd.cluster.local ca.crt ca.key \
  2.   --profile root-ca --no-password --insecure --san webhook.linkerd.cluster.local &&
  3.   kubectl create secret tls \
  4.    webhook-issuer-tls \
  5.    --cert=ca.crt \
  6.    --key=ca.key \
  7.    --namespace=linkerd

Create an Issuer referencing the secret

With the Secret in place, we can create a cert-manager “Issuer” resource that references it:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1alpha3
  3. kind: Issuer
  4. metadata:
  5.   name: webhook-issuer
  6.   namespace: linkerd
  7. spec:
  8.   ca:
  9.     secretName: webhook-issuer-tls
  10. EOF

Issuing certificates and writing them to a secret

Finally, we can create cert-manager “Certificate” resources which use the Issuer to generate the desired certificates:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1alpha3
  3. kind: Certificate
  4. metadata:
  5.   name: linkerd-proxy-injector
  6.   namespace: linkerd
  7. spec:
  8.   secretName: linkerd-proxy-injector-k8s-tls
  9.   duration: 24h
  10.   renewBefore: 1h
  11.   issuerRef:
  12.     name: webhook-issuer
  13.     kind: Issuer
  14.   commonName: linkerd-proxy-injector.linkerd.svc
  15.   isCA: false
  16.   keyAlgorithm: ecdsa
  17.   usages:
  18.   - server auth
  19. ---
  20. apiVersion: cert-manager.io/v1alpha3
  21. kind: Certificate
  22. metadata:
  23.   name: linkerd-sp-validator
  24.   namespace: linkerd
  25. spec:
  26.   secretName: linkerd-sp-validator-k8s-tls
  27.   duration: 24h
  28.   renewBefore: 1h
  29.   issuerRef:
  30.     name: webhook-issuer
  31.     kind: Issuer
  32.   commonName: linkerd-sp-validator.linkerd.svc
  33.   isCA: false
  34.   keyAlgorithm: ecdsa
  35.   usages:
  36.   - server auth
  37. ---
  38. apiVersion: cert-manager.io/v1alpha3
  39. kind: Certificate
  40. metadata:
  41.   name: linkerd-tap
  42.   namespace: linkerd
  43. spec:
  44.   secretName: linkerd-tap-k8s-tls
  45.   duration: 24h
  46.   renewBefore: 1h
  47.   issuerRef:
  48.     name: webhook-issuer
  49.     kind: Issuer
  50.   commonName: linkerd-tap.linkerd.svc
  51.   isCA: false
  52.   keyAlgorithm: ecdsa
  53.   usages:
  54.   - server auth
  55. EOF

At this point, cert-manager can now use these Certificate resources to obtain TLS credentials, which are stored in the linkerd-proxy-injector-k8s-tls, linkerd-sp-validator-k8s-tls, and linkerd-tap-k8s-tls secrets respectively.

Now we just need to inform Linkerd to consume these credentials.

Using these credentials with CLI installation

To configure Linkerd to use the credentials from cert-manager rather than generating its own, we generate a supplemental config file:

  1. CA=$(awk '{ print "    " $0 }' ca.crt); cat > config.yml <<EOF
  2. proxyInjector:
  3.   externalSecret: true
  4.   caBundle: |
  5. $CA
  6. profileValidator:
  7.   externalSecret: true
  8.   caBundle: |
  9. $CA
  10. tap:
  11.   externalSecret: true
  12.   caBundle: |
  13. $CA
  14. EOF

Now we can install Linkerd using this config file:

  1. linkerd install --config=config.yml | kubectl apply -f -

Installing with Helm

For Helm installation, we can configure the Helm values directly:

  1. helm install linkerd2 \
  2.   --set installNamespace=false \
  3.   --set proxyInjector.externalSecret=true \
  4.   --set-file proxyInjector.caBundle=ca.crt \
  5.   --set profileValidator.externalSecret=true \
  6.   --set-file profileValidator.caBundle=ca.crt \
  7.   --set tap.externalSecret=true \
  8.   --set-file tap.caBundle=ca.crt \
  9.   linkerd/linkerd2 \
  10.   -n linkerd

Note

When installing Linkerd with Helm, you must also provide the issuer trust root and issuer credentials as described in Installing Linkerd with Helm.

Note

For Helm versions < v3, --name flag has to specifically be passed. In Helm v3, It has been deprecated, and is the first argument as specified above.