Automatically Rotating Webhook TLS Credentials

The Linkerd control plane contains several components, called webhooks, which are called directly by Kubernetes itself. The traffic from Kubernetes to the Linkerd webhooks is secured with TLS and therefore each of the webhooks requires a secret containing TLS credentials. These certificates are different from the ones that the Linkerd proxies use to secure pod-to-pod communication and use a completely separate trust chain. For more information on rotating the TLS credentials used by the Linkerd proxies, see Automatically Rotating Control Plane TLS Credentials.

By default, when Linkerd is installed with the Linkerd CLI or with the Linkerd Helm chart, TLS credentials are automatically generated for all of the webhooks. If these certificates expire or need to be regenerated for any reason, performing a Linkerd upgrade (using the Linkerd CLI or using Helm) will regenerate them.

This workflow is suitable for most users. However, if you need these webhook certificates to be rotated automatically on a regular basis, it is possible to use cert-manager to automatically manage them.

Install Cert manager

As a first step, install cert-manager on your cluster and create the namespaces that cert-manager will use to store its webhook-related resources. For simplicity, we suggest using the defaule namespace linkerd uses:

  1. # control plane core
  2. kubectl create namespace linkerd
  4. # viz (ignore if not using the viz extension)
  5. kubectl create namespace linkerd-viz
  7. # viz (ignore if not using the jaeger extension)
  8. kubectl create namespace linkerd-jaeger

Save the signing key pair as a Secret

Next, we will use the step tool, to create a signing key pair which will be used to sign each of the webhook certificates:

  1. step certificate create webhook.linkerd.cluster.local ca.crt ca.key \
  2.   --profile root-ca --no-password --insecure --san webhook.linkerd.cluster.local
  4. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd
  6. # ignore if not using the viz extension
  7. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-viz
  9. # ignore if not using the jaeger extension
  10. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-jaeger

Create Issuers referencing the secrets

With the Secrets in place, we can create cert-manager “Issuer” resources that reference them:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1
  3. kind: Issuer
  4. metadata:
  5.   name: webhook-issuer
  6.   namespace: linkerd
  7. spec:
  8.   ca:
  9.     secretName: webhook-issuer-tls
  10. ---
  11. # ignore if not using the viz extension
  12. apiVersion: cert-manager.io/v1
  13. kind: Issuer
  14. metadata:
  15.   name: webhook-issuer
  16.   namespace: linkerd-viz
  17. spec:
  18.   ca:
  19.     secretName: webhook-issuer-tls
  20. ---
  21. # ignore if not using the jaeger extension
  22. apiVersion: cert-manager.io/v1
  23. kind: Issuer
  24. metadata:
  25.   name: webhook-issuer
  26.   namespace: linkerd-jaeger
  27. spec:
  28.   ca:
  29.     secretName: webhook-issuer-tls
  30. EOF

Issuing certificates and writing them to secrets

Finally, we can create cert-manager “Certificate” resources which use the Issuers to generate the desired certificates:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1
  3. kind: Certificate
  4. metadata:
  5.   name: linkerd-proxy-injector
  6.   namespace: linkerd
  7. spec:
  8.   secretName: linkerd-proxy-injector-k8s-tls
  9.   duration: 24h
  10.   renewBefore: 1h
  11.   issuerRef:
  12.     name: webhook-issuer
  13.     kind: Issuer
  14.   commonName: linkerd-proxy-injector.linkerd.svc
  15.   dnsNames:
  16.   - linkerd-proxy-injector.linkerd.svc
  17.   isCA: false
  18.   privateKey:
  19.     algorithm: ECDSA
  20.   usages:
  21.   - server auth
  22. ---
  23. apiVersion: cert-manager.io/v1
  24. kind: Certificate
  25. metadata:
  26.   name: linkerd-sp-validator
  27.   namespace: linkerd
  28. spec:
  29.   secretName: linkerd-sp-validator-k8s-tls
  30.   duration: 24h
  31.   renewBefore: 1h
  32.   issuerRef:
  33.     name: webhook-issuer
  34.     kind: Issuer
  35.   commonName: linkerd-sp-validator.linkerd.svc
  36.   dnsNames:
  37.   - linkerd-sp-validator.linkerd.svc
  38.   isCA: false
  39.   privateKey:
  40.     algorithm: ECDSA
  41.   usages:
  42.   - server auth
  43. ---
  44. # ignore if not using the viz extension
  45. apiVersion: cert-manager.io/v1
  46. kind: Certificate
  47. metadata:
  48.   name: tap
  49.   namespace: linkerd-viz
  50. spec:
  51.   secretName: tap-k8s-tls
  52.   duration: 24h
  53.   renewBefore: 1h
  54.   issuerRef:
  55.     name: webhook-issuer
  56.     kind: Issuer
  57.   commonName: tap.linkerd-viz.svc
  58.   dnsNames:
  59.   - tap.linkerd-viz.svc
  60.   isCA: false
  61.   privateKey:
  62.     algorithm: ECDSA
  63.   usages:
  64.   - server auth
  65. ---
  66. # ignore if not using the viz extension
  67. apiVersion: cert-manager.io/v1
  68. kind: Certificate
  69. metadata:
  70.   name: linkerd-tap-injector
  71.   namespace: linkerd-viz
  72. spec:
  73.   secretName: tap-injector-k8s-tls
  74.   duration: 24h
  75.   renewBefore: 1h
  76.   issuerRef:
  77.     name: webhook-issuer
  78.     kind: Issuer
  79.   commonName: tap-injector.linkerd-viz.svc
  80.   dnsNames:
  81.   - tap-injector.linkerd-viz.svc
  82.   isCA: false
  83.   privateKey:
  84.     algorithm: ECDSA
  85.   usages:
  86.   - server auth
  87. ---
  88. # ignore if not using the jaeger extension
  89. apiVersion: cert-manager.io/v1
  90. kind: Certificate
  91. metadata:
  92.   name: jaeger-injector
  93.   namespace: linkerd-jaeger
  94. spec:
  95.   secretName: jaeger-injector-k8s-tls
  96.   duration: 24h
  97.   renewBefore: 1h
  98.   issuerRef:
  99.     name: webhook-issuer
  100.     kind: Issuer
  101.   commonName: jaeger-injector.linkerd.svc
  102.   dnsNames:
  103.   - jaeger-injector.linkerd.svc
  104.   isCA: false
  105.   privateKey:
  106.     algorithm: ECDSA
  107.   usages:
  108.   - server auth
  109. EOF

At this point, cert-manager can now use these Certificate resources to obtain TLS credentials, which are stored in the linkerd-proxy-injector-k8s-tls, linkerd-sp-validator-k8s-tls, tap-k8s-tls, tap-injector-k8s-tls and jaeger-injector-k8s-tls secrets respectively.

Now we just need to inform Linkerd to consume these credentials.

Using these credentials with CLI installation

To configure Linkerd to use the credentials from cert-manager rather than generating its own, we generate a supplemental config file:

  1. CA=$(awk '{ print "    " $0 }' ca.crt)
  3. cat > config.yml <<EOF
  4. proxyInjector:
  5.   externalSecret: true
  6.   caBundle: |
  7. $CA
  8. profileValidator:
  9.   externalSecret: true
  10.   caBundle: |
  11. $CA
  12. EOF
  14. # ignore if not using the viz extension
  15. cat > config-viz.yml <<EOF
  16. tap:
  17.   externalSecret: true
  18.   caBundle: |
  19. $CA
  20. tapInjector:
  21.   externalSecret: true
  22.   caBundle: |
  23. $CA
  24. EOF
  26. # ignore if not using the jaeger extension
  27. cat > config-jaeger.yml <<EOF
  28. webhook:
  29.   externalSecret: true
  30.   caBundle: |
  31. $CA
  32. EOF

Now we can install Linkerd using these config files:

  1. linkerd install --values=config.yml | kubectl apply -f -
  3. # ignore if not using the viz extension
  4. linkerd viz install --values=config-viz.yml | kubectl apply -f -
  6. # ignore if not using the jaeger extension
  7. linkerd jaeger install --values=config-jaeger.yml | kubectl apply -f -

Installing with Helm

For Helm installation, we can configure the Helm values directly:

  1. helm install linkerd2 \
  2.   --set installNamespace=false \
  3.   --set proxyInjector.externalSecret=true \
  4.   --set-file proxyInjector.caBundle=ca.crt \
  5.   --set profileValidator.externalSecret=true \
  6.   --set-file profileValidator.caBundle=ca.crt \
  7.   linkerd/linkerd2 \
  8.   -n linkerd
  10. # ignore if not using the viz extension
  11. helm install linkerd-viz \
  12.   --set installNamespace=false \
  13.   --set tap.externalSecret=true \
  14.   --set-file tap.caBundle=ca.crt \
  15.   --set tapInjector.externalSecret=true \
  16.   --set-file tapInjector.caBundle=ca.crt \
  17.   linkerd/linkerd-viz \
  18.   -n linkerd-viz
  20. # ignore if not using the jaeger extension
  21. helm install linkerd-jaeger \
  22.   --set installNamespace=false \
  23.   --set webhook.externalSecret=true \
  24.   --set-file webhook.caBundle=ca.crt \
  25.   linkerd/linkerd-jaeger \
  26.   -n linkerd-jaeger

Note

When installing Linkerd with Helm, you must also provide the issuer trust root and issuer credentials as described in Installing Linkerd with Helm.

Note

For Helm versions < v3, --name flag has to specifically be passed. In Helm v3, It has been deprecated, and is the first argument as specified above.

See Automatically Rotating Control Plane TLS Credentials for details on how to do something similar for control plane credentials.