Mutual TLS

This policy enables automatic encrypted mTLS traffic for all the services in a Mesh, as well as assigning an identity to every data plane proxy. Kuma supports different types of CA backends as well as automatic certificate rotation.

Kuma ships with the following CA (Certificate Authority) supported backends:

  • builtin: it automatically auto-generates a CA root certificate and key, that are also being automatically stored as a Secret.
  • provided: the CA root certificate and key are being provided by the user in the form of a Secret.

Once a CA backend has been specified, Kuma will then automatically generate a certificate for every data plane proxy in the Mesh. The data plane certificates that Kuma generates are SPIFFE compatible and they are being used for AuthN/Z use-cases in order to identify every workload in our system.

The certificates that Kuma generates have a SAN set to spiffe://<mesh name>/<service name>. When Kuma enforces policies that require an identity like TrafficPermission it will extract the SAN from the client certificate and use it to match the service identity.

Remember that by default mTLS is not enabled and needs to be explicitly enabled as described below. Also remember that by default when mTLS is enabled all traffic is denied unless a TrafficPermission policy is being configured to explicitly allow traffic across our data planes.

Always create a TrafficPermission resource before enabling mTLS in a Mesh in order to avoid unexpected traffic interruptions caused by a lack of authorization among the data planes.

To enable mTLS we need to configure the mtls property in a Mesh resource. We can have as many backends as we want, but only one at a time can be enabled via the enabledBackend property.

If enabledBackend is missing or empty, then mTLS will be disabled for the entire Mesh.

Usage of “builtin” CA

This is the fastest and simplest way to enable mTLS in Kuma.

With a builtin CA backend type, Kuma will dynamically generate its own CA root certificate and key that it will use to automatically provision (and rotate) data plane certificates for every replica of every service.

We can specify more than one builtin backend with different names, and each one of them will be automatically provisioned with a unique pair of certificate + key (they are not shared).

To enable a builtin mTLS for the entire Mesh we can apply the following configuration:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   mtls:
  7.     enabledBackend: ca-1
  8.     backends:
  9.       - name: ca-1
  10.         type: builtin
  11.         dpCert:
  12.           rotation:
  13.             expiration: 1d
  14.         conf:
  15.           caCert:
  16.             RSAbits: 2048
  17.             expiration: 10y

We will apply the configuration with kubectl apply -f [..].

  1. type: Mesh
  2. name: default
  3. mtls:
  4.   enabledBackend: ca-1
  5.   backends:
  6.     - name: ca-1
  7.       type: builtin
  8.       dpCert:
  9.         rotation:
  10.           expiration: 1d
  11.       conf:
  12.         caCert:
  13.           RSAbits: 2048
  14.           expiration: 10y

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

A few considerations:

  • The dpCert configuration determines how often Kuma should automatically rotate the certificates assigned to every data plane proxy.
  • The caCert configuration determines a few properties that Kuma will use when auto-generating the CA root certificate.

Storage of Secrets

When using a builtin backend Kuma automatically generates a root CA certificate and key that are being stored as a Kuma Secret resource with the following name:

  • {mesh name}.ca-builtin-cert-{backend name} for the certificate
  • {mesh name}.ca-builtin-key-{backend name} for the key

On Kubernetes, Kuma secrets are being stored in the kuma-system namespace, while on Universal they are being stored in the underlying backend configured in kuma-cp.

We can retrieve the secrets via kumactl on both Universal and Kubernetes, or via kubectl on Kubernetes only:

The following command can be executed on any Kuma backend:

  1. $ kumactl get secrets [-m MESH]
  2. MESH      NAME                           AGE
  3. default   default.ca-builtin-cert-ca-1   1m
  4. default   default.ca-builtin-key-ca-1    1m

The following command can be executed only on Kubernetes:

  1. $ kubectl get secrets \
  2.     -n kuma-system \
  3.     --field-selector='type=system.kuma.io/secret'
  4. NAME                             TYPE                                  DATA   AGE
  5. default.ca-builtin-cert-ca-1     system.kuma.io/secret                 1      1m
  6. default.ca-builtin-key-ca-1      system.kuma.io/secret                 1      1m

Usage of “provided” CA

Whenever we want to provide our own CA root certificate and key, we can use the provided backend.

Unlike the builtin backend, with provided we need to first upload the certificate and key that Kuma will use as Secret resources, and then reference the Secrets to the mTLS configuration.

We are responsible for providing the CA root certificate + key and also to manage their lifecycle. Kuma will then use our CA root certificate + Key to automatically provision (and rotate) data plane certificates for every replica of every service.

First we need to upload our CA root certificate and key as Kuma Secrets so that we can later reference them.

Once the secrets have been created, to enable a provided mTLS for the entire Mesh we can apply the following configuration:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   mtls:
  7.     enabledBackend: ca-1
  8.     backends:
  9.       - name: ca-1
  10.         type: provided
  11.         dpCert:
  12.           rotation:
  13.             expiration: 1d
  14.         conf:
  15.           cert:
  16.             secret: name-of-secret
  17.           key:
  18.             secret: name-of-secret

We will apply the configuration with kubectl apply -f [..].

  1. type: Mesh
  2. name: default
  3. mtls:
  4.   enabledBackend: ca-1
  5.   backends:
  6.     - name: ca-1
  7.       type: provided
  8.       dpCert:
  9.         rotation:
  10.           expiration: 1d
  11.       conf:
  12.         cert:
  13.           secret: name-of-secret
  14.         key:
  15.           secret: name-of-secret

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

A few considerations:

  • The dpCert configuration determines how often Kuma should automatically rotate the certificates assigned to every data plane proxy.
  • The Secrets must exist before referencing them in a provided backend.

CA requirements

When using an arbitrary certificate and key for a provided backend, we must make sure that we comply with the following requirements:

  1. It MUST be a self-signed Root CA certificate (Intermediate CA certificates are not allowed)
  2. It MUST have basic constraint CA set to true (see X509-SVID: 4.1. Basic ConstraintsMutual TLS - 图1)
  3. It MUST have key usage extension keyCertSign set (see X509-SVID: 4.3. Key UsageMutual TLS - 图2)
  4. It MUST NOT have key usage extension ‘digitalSignature’ set (see X509-SVID: Appendix A. X.509 Field ReferenceMutual TLS - 图3)
  5. It MUST NOT have key usage extension ‘keyAgreement’ set (see X509-SVID: Appendix A. X.509 Field ReferenceMutual TLS - 图4)
  6. It MUST NOT have key usage extension ‘keyEncipherment’ set (see X509-SVID: Appendix A. X.509 Field ReferenceMutual TLS - 图5)

Do not use the following example in production, instead generate valid and compliant certificates. This example is intended for usage in a development environment.

Below we can find an example to generate a sample CA certificate + key:

The following command will generate a CA root certificate and key that can be uploaded to Kuma as a Secret and then used in a provided mTLS backend:

  1. SAMPLE_CA_CONFIG="
  2. [req]
  3. distinguished_name=dn
  4. [ dn ]
  5. [ ext ]
  6. basicConstraints=CA:TRUE,pathlen:0
  7. keyUsage=keyCertSign
  8. "
  10. openssl req -config <(echo "$SAMPLE_CA_CONFIG") -new -newkey rsa:2048 -nodes \
  11.   -subj "/CN=Hello" -x509 -extensions ext -keyout key.pem -out crt.pem

The command will generate a certificate at crt.pem and the key at key.pem. We can generate the Kuma Secret resources by following the Secret reference.

Development Mode

In development mode we may want to provide the cert and key properties of the provided backend without necessarily having to create a Secret resource, but by using either a file or an inline value.

Using the file and inline modes in production presents a security risk since it makes the values of our CA root certificate and key more easily accessible from a malicious actor. We highly recommend using file and inline only in development mode.

Kuma offers an alternative way to specify the CA root certificate and key:

Please note the file and inline properties that are being used instead of secret:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: Mesh
  3. metadata:
  4.   name: default
  5. spec:
  6.   mtls:
  7.     enabledBackend: ca-1
  8.     backends:
  9.     - name: ca-1
  10.       type: provided
  11.       config:
  12.         cert:
  13.           inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # cert in Base64
  14.         key:
  15.           file: /opt/cert.key

Please note the file and inline properties that are being used instead of secret:

  1. type: Mesh
  2. name: default
  3. mtls:
  4.   enabledBackend: ca-1
  5.   backends:
  6.   - name: ca-1
  7.     type: provided
  8.     config:
  9.       cert:
  10.         inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHekNDQWdPZ0F3S... # cert in Base64
  11.       key:
  12.         file: /opt/cert.key

Certificate Rotation

Once a CA backend has been configured, Kuma will utilize the CA root certificate and key to automatically provision a certificate for every data plane proxy that it connects to kuma-cp.

Unlike the CA certificate, the data plane proxy certificates are not permanently stored anywhere but they only reside in memory. Data plane certificates are designed to be short-lived and rotated often by Kuma.

By default, the expiration time of a data plane certificate is 30 days. Kuma rotates data plane certificates automatically after 4/5 of the certificate validity time (ie: for the default 30 days expiration, that would be every 24 days).

We can update the duration of the data plane certificates by updating the dpCert property - as documented in this page - on every mTLS backend available.

Finally, we can inspect the certificate rotation statistics by executing the following command (supported on both Kubernetes and Universal):

We can use the Kuma CLI:

  1. $ kumactl inspect dataplanes
  2. MESH      NAME     TAGS          STATUS   LAST CONNECTED AGO   LAST UPDATED AGO   TOTAL UPDATES   TOTAL ERRORS   CERT REGENERATED AGO   CERT EXPIRATION       CERT REGENERATIONS
  3. default   web-01   service=web   Online   5s                   3s                 4               0              3s                     2020-05-11 16:01:34   2

Please note the CERT REGENERATED AGO, CERT EXPIRATION, CERT REGENERATIONS columns.

We can use the Kuma HTTP API by retrieving the Dataplane Insight resource and inspecting the dataplaneInsight object.

  1. ...
  2. dataplaneInsight": {
  3.   ...
  4.   "mTLS": {
  5.     "certificateExpirationTime": "2020-05-14T20:15:23Z",
  6.     "lastCertificateRegeneration": "2020-05-13T20:15:23.994549539Z",
  7.     "certificateRegenerations": 1
  8.   }
  9. }
  10. ...

A new data plane certificate will be automatically (re)generated when:

  • A data plane proxy is being restarted.
  • The control plane is being restarted.
  • The data plane proxy connects to a new control plane.