Encrypting Secret Data at Rest

This page shows how to enable and configure encryption of secret data at rest.

Before you begin

  • You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

    Your Kubernetes server must be at or later than version 1.13. To check the version, enter kubectl version.

  • etcd v3.0 or later is required

  • To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.

  • Use of wildcard for resource encryption is available from Kubernetes v1.27 or newer.

Configuration and determining whether encryption at rest is already enabled

The kube-apiserver process accepts an argument --encryption-provider-config that controls how API data is encrypted in etcd. The configuration is provided as an API named EncryptionConfiguration. --encryption-provider-config-automatic-reload boolean argument determines if the file set by --encryption-provider-config should be automatically reloaded if the disk contents change. This enables key rotation without API server restarts. An example configuration is provided below.

Caution: IMPORTANT: For high-availability configurations (with two or more control plane nodes), the encryption configuration file must be the same! Otherwise, the kube-apiserver component cannot decrypt data stored in the etcd.

Understanding the encryption at rest configuration

  1. apiVersion: apiserver.config.k8s.io/v1
  2. kind: EncryptionConfiguration
  3. resources:
  4. - resources:
  5. - secrets
  6. - configmaps
  7. - pandas.awesome.bears.example
  8. providers:
  9. - identity: {}
  10. - aesgcm:
  11. keys:
  12. - name: key1
  13. secret: c2VjcmV0IGlzIHNlY3VyZQ==
  14. - name: key2
  15. secret: dGhpcyBpcyBwYXNzd29yZA==
  16. - aescbc:
  17. keys:
  18. - name: key1
  19. secret: c2VjcmV0IGlzIHNlY3VyZQ==
  20. - name: key2
  21. secret: dGhpcyBpcyBwYXNzd29yZA==
  22. - secretbox:
  23. keys:
  24. - name: key1
  25. secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
  26. - resources:
  27. - events
  28. providers:
  29. - identity: {} # do not encrypt events even though *.* is specified below
  30. - resources:
  31. - '*.apps'
  32. providers:
  33. - aescbc:
  34. keys:
  35. - name: key2
  36. secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgaXQ/Cg==
  37. - resources:
  38. - '*.*'
  39. providers:
  40. - aescbc:
  41. keys:
  42. - name: key3
  43. secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==

Each resources array item is a separate config and contains a complete configuration. The resources.resources field is an array of Kubernetes resource names (resource or resource.group) that should be encrypted like Secrets, ConfigMaps, or other resources.

If custom resources are added to EncryptionConfiguration and the cluster version is 1.26 or newer, any newly created custom resources mentioned in the EncryptionConfiguration will be encrypted. Any custom resources that existed in etcd prior to that version and configuration will be unencrypted until they are next written to storage. This is the same behavior as built-in resources. See the Ensure all secrets are encrypted section.

The providers array is an ordered list of the possible encryption providers to use for the APIs that you listed.

Only one provider type may be specified per entry (identity or aescbc may be provided, but not both in the same item). The first provider in the list is used to encrypt resources written into the storage. When reading resources from storage, each provider that matches the stored data attempts in order to decrypt the data. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource.

EncryptionConfiguration supports the use of wildcards to specify the resources that should be encrypted. Use ‘*.<group>‘ to encrypt all resources within a group (for eg ‘*.apps‘ in above example) or ‘*.*‘ to encrypt all resources. ‘*.‘ can be used to encrypt all resource in the core group. ‘*.*‘ will encrypt all resources, even custom resources that are added after API server start.

Note: Use of wildcards that overlap within the same resource list or across multiple entries are not allowed since part of the configuration would be ineffective. The resources list’s processing order and precedence are determined by the order it’s listed in the configuration.

Opting out of encryption for specific resources while wildcard is enabled can be achieved by adding a new resources array item with the resource name, followed by the providers array item with the identity provider. For example, if ‘*.*‘ is enabled and you want to opt-out encryption for the events resource, add a new item to the resources array with events as the resource name, followed by the providers array item with identity. The new item should look like this:

  1. - resources:
  2. - events
  3. providers:
  4. - identity: {}

Ensure that the new item is listed before the wildcard ‘*.*‘ item in the resources array to give it precedence.

For more detailed information about the EncryptionConfiguration struct, please refer to the encryption configuration API.

Caution: If any resource is not readable via the encryption config (because keys were changed), the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to read that resource will fail until it is deleted or a valid decryption key is provided.

Providers

Providers for Kubernetes encryption at rest
NameEncryptionStrengthSpeedKey LengthOther Considerations
identityNoneN/AN/AN/AResources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
secretboxXSalsa20 and Poly1305StrongFaster32-byteA newer standard and may not be considered acceptable in environments that require high levels of review.
aesgcmAES-GCM with random nonceMust be rotated every 200k writesFastest16, 24, or 32-byteIs not recommended for use except when an automated key rotation scheme is implemented.
aescbcAES-CBC with PKCS#7 paddingWeakFast32-byteNot recommended due to CBC’s vulnerability to padding oracle attacks.
kms v1Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding (prior to v1.25), using AES-GCM starting from v1.25, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS)StrongestSlow (compared to kms v2)32-bytesSimplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. Configure the KMS V1 provider.
kms v2Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-GCM, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS)StrongestFast32-bytesThe recommended choice for using a third party tool for key management. Available in beta from v1.27. A new DEK is generated at startup and reused for encryption. The DEK is rotated when the KEK is rotated. Configure the KMS V2 provider.

Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider is the first provider, the first key is used for encryption.

Caution: Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. Please use kms provider for additional security.

By default, the identity provider is used to protect secret data in etcd, which provides no encryption. EncryptionConfiguration was introduced to encrypt secret data locally, with a locally managed key.

Encrypting secret data with a locally managed key protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption keys.

Envelope encryption creates dependence on a separate key, not stored in Kubernetes. In this case, an attacker would need to compromise etcd, the kubeapi-server, and the third-party KMS provider to retrieve the plaintext values, providing a higher level of security than locally stored encryption keys.

Encrypting your data

Create a new encryption config file:

  1. apiVersion: apiserver.config.k8s.io/v1
  2. kind: EncryptionConfiguration
  3. resources:
  4. - resources:
  5. - secrets
  6. - configmaps
  7. - pandas.awesome.bears.example
  8. providers:
  9. - aescbc:
  10. keys:
  11. - name: key1
  12. secret: <BASE 64 ENCODED SECRET>
  13. - identity: {}

To create a new Secret, perform the following steps:

  1. Generate a 32-byte random key and base64 encode it. If you’re on Linux or macOS, run the following command:

    1. head -c 32 /dev/urandom | base64
  2. Place that value in the secret field of the EncryptionConfiguration struct.

  3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.

    You will need to mount the new encryption config file to the kube-apiserver static pod. Here is an example on how to do that:

    1. Save the new encryption config file to /etc/kubernetes/enc/enc.yaml on the control-plane node.
    2. Edit the manifest for the kube-apiserver static pod: /etc/kubernetes/manifests/kube-apiserver.yaml similarly to this:
    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. annotations:
    5. kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.10.30.4:6443
    6. creationTimestamp: null
    7. labels:
    8. component: kube-apiserver
    9. tier: control-plane
    10. name: kube-apiserver
    11. namespace: kube-system
    12. spec:
    13. containers:
    14. - command:
    15. - kube-apiserver
    16. ...
    17. - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml # <-- add this line
    18. volumeMounts:
    19. ...
    20. - name: enc # <-- add this line
    21. mountPath: /etc/kubernetes/enc # <-- add this line
    22. readonly: true # <-- add this line
    23. ...
    24. volumes:
    25. ...
    26. - name: enc # <-- add this line
    27. hostPath: # <-- add this line
    28. path: /etc/kubernetes/enc # <-- add this line
    29. type: DirectoryOrCreate # <-- add this line
    30. ...
  4. Restart your API server.

Caution: Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict permissions on your control-plane nodes so only the user who runs the kube-apiserver can read it.

Verifying that data is encrypted

Data is encrypted when written to etcd. After restarting your kube-apiserver, any newly created or updated Secret or other resource types configured in EncryptionConfiguration should be encrypted when stored. To check this, you can use the etcdctl command line program to retrieve the contents of your secret data.

  1. Create a new Secret called secret1 in the default namespace:

    1. kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
  2. Using the etcdctl command line, read that Secret out of etcd:

    1. ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C

    where [...] must be the additional arguments for connecting to the etcd server.

    For example:

    1. ETCDCTL_API=3 etcdctl \
    2. --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    3. --cert=/etc/kubernetes/pki/etcd/server.crt \
    4. --key=/etc/kubernetes/pki/etcd/server.key \
    5. get /registry/secrets/default/secret1 | hexdump -C

    The output is similar to this (abbreviated):

    1. 00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
    2. 00000010 73 2f 64 65 66 61 75 6c 74 2f 73 65 63 72 65 74 |s/default/secret|
    3. 00000020 31 0a 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 |1.k8s:enc:aescbc|
    4. 00000030 3a 76 31 3a 6b 65 79 31 3a c7 6c e7 d3 09 bc 06 |:v1:key1:.l.....|
    5. 00000040 25 51 91 e4 e0 6c e5 b1 4d 7a 8b 3d b9 c2 7c 6e |%Q...l..Mz.=..|n|
    6. 00000050 b4 79 df 05 28 ae 0d 8e 5f 35 13 2c c0 18 99 3e |.y..(..._5.,...>|
    7. [...]
    8. 00000110 23 3a 0d fc 28 ca 48 2d 6b 2d 46 cc 72 0b 70 4c |#:..(.H-k-F.r.pL|
    9. 00000120 a5 fc 35 43 12 4e 60 ef bf 6f fe cf df 0b ad 1f |..5C.N`..o......|
    10. 00000130 82 c4 88 53 02 da 3e 66 ff 0a |...S..>f..|
    11. 0000013a
  3. Verify the stored Secret is prefixed with k8s:enc:aescbc:v1: which indicates the aescbc provider has encrypted the resulting data. Confirm that the key name shown in etcd matches the key name specified in the EncryptionConfiguration mentioned above. In this example, you can see that the encryption key named key1 is used in etcd and in EncryptionConfiguration.

  4. Verify the Secret is correctly decrypted when retrieved via the API:

    1. kubectl get secret secret1 -n default -o yaml

    The output should contain mykey: bXlkYXRh, with contents of mydata encoded, check decoding a Secret to completely decode the Secret.

Ensure all Secrets are encrypted

Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.

  1. kubectl get secrets --all-namespaces -o json | kubectl replace -f -

The command above reads all Secrets and then updates them to apply server side encryption.

Note: If an error occurs due to a conflicting write, retry the command. For larger clusters, you may wish to subdivide the secrets by namespace or script an update.

Rotating a decryption key

Changing a Secret without incurring downtime requires a multi-step operation, especially in the presence of a highly-available deployment where multiple kube-apiserver processes are running.

  1. Generate a new key and add it as the second key entry for the current provider on all servers
  2. Restart all kube-apiserver processes to ensure each server can decrypt using the new key
  3. Make the new key the first entry in the keys array so that it is used for encryption in the config
  4. Restart all kube-apiserver processes to ensure each server now encrypts using the new key
  5. Run kubectl get secrets --all-namespaces -o json | kubectl replace -f - to encrypt all existing Secrets with the new key
  6. Remove the old decryption key from the config after you have backed up etcd with the new key in use and updated all Secrets

When running a single kube-apiserver instance, step 2 may be skipped.

Decrypting all data

To disable encryption at rest, place the identity provider as the first entry in the config and restart all kube-apiserver processes.

  1. apiVersion: apiserver.config.k8s.io/v1
  2. kind: EncryptionConfiguration
  3. resources:
  4. - resources:
  5. - secrets
  6. providers:
  7. - identity: {}
  8. - aescbc:
  9. keys:
  10. - name: key1
  11. secret: <BASE 64 ENCODED SECRET>

Then run the following command to force decrypt all Secrets:

  1. kubectl get secrets --all-namespaces -o json | kubectl replace -f -

What’s next