Keyring and Data Encryption

Kong Gateway provides a mechanism to store sensitive data fields, such as consumer secrets, in an encrypted format within the database. This provides for encryption-at-rest security controls in a Kong cluster.

This functionality provides transparent, symmetric encryption of sensitive data fields at rest. Transparency refers to the fact that, when enabled, encryption/decryption of data is done on-the-fly by Kong immediately before writing/immediately after reading from the database. Responses generated by the Admin API containing sensitive fields continue to show data as plaintext, and runtime elements of Kong (such as plugins) that require access to sensitive fields do so transparently, without requiring additional configuration.

Getting Started

This document provides a brief introduction to leveraging Kong Gateway symmetric database encryption to store sensitive configuration data in a Kong cluster. This document covers necessary Kong configuration settings and key management lifecycle procedures when using database encryption in cluster mode. This mode offers the simplest method to getting started, as it requires no external dependencies.

Generate a Management RSA Key Pair

Before enabling database encryption, we strongly recommend you generate an RSA key pair for use in exporting and recovering keyring material. If keyring material is lost, it is impossible to recover any sensitive data fields written to the database with a key on that keyring.

Generating an RSA key pair is straightforward via the openssl CLI:

  1. $ openssl genrsa -out key.pem 2048
  2. $ openssl rsa -in key.pem -pubout -out cert.pem

This key pair will be provided to Kong in order to faciliate exporting and re-importing the keyring. The public cert.pem and private key.pem should be stored securely in accordance with your existing secrets management policies. Details on secure storage of RSA key pairs is outside the scope of this documentation.

Configure Kong for database encryption

Enabling data encryption in Kong requires modifying the Kong configuration. Set the following values in kong.conf:

  1. keyring_enabled = on
  2. keyring_strategy = cluster
  3. keyring_public_key = /path/to/generated/cert.pem
  4. keyring_private_key = /path/to/generated/key.pem

Or via environmental variables:

  1. export KONG_KEYRING_ENABLED=on
  2. export KONG_KEYRING_STRATEGY=cluster
  3. export KONG_KEYRING_PUBLIC_KEY=/path/to/generated/cert.pem
  4. export KONG_KEYRING_PRIVATE_KEY=/path/to/generated/key.pem

All nodes in the Kong cluster should share the same keyring_enabled and keyring_strategy configuration values. Not every node needs to be provided the management RSA key pair, as that key pair is only used for backup and recovery processes. It does not need to be present for regular database read/write operations.

Note that the user under which Kong worker processes run must have read access to the public and private keys in order to be able to perform keyring export and import operations. This user is defined by the nginx_user Kong configuration option. We recommend restricting access to these files as tightly as possible. For example:

  1. $ chown <nginx_user>:<nginx_user> /path/to/generated/cert.pem /path/to/generated/key.pem
  2. $ chmod 400 /path/to/generated/cert.pem /path/to/generated/key.pem

When testing, you can also set keyring_blob_path in kong.conf or KONG_KEYRING_BLOB_PATH using environmental variables to specify a path to dump known keys. The dumped keys are encrypted with the public RSA key defined in the keyring_public_key Kong configuration value, and are automatically loaded during Kong start.

Start Kong

Once all Kong nodes in the cluster have been configured, start each node:

  1. $ kong start

When encryption is enabled on a Kong node, it checks the status of the cluster keyring on boot. If it detects that no keys are present in the keyring, it will generate a key automatically. This process allows encryption/decryption operations to begin immediately.

For all other nodes, the generated key will be automatically distributed within a few seconds.

Note that encryption keys are held only in memory, and do not persist past a restart of the Kong process (e.g., running kong restart). Because of this limitation, you must export the keyring following its initialization. Otherwise, if all Kong nodes in a cluster restart simultaneously, any sensitive fields written with the keyring become unrecoverable. Key material still persists after a soft reload of Kong (i.e., kong reload).

Verify the Cluster Keyring

With the keyring enabled and Kong started, verify the contents of the keyring:

  1. $ curl -s localhost:8001/keyring | jq
  2. {
  3.   "ids": [
  4.     "LaW1urRQ"
  5.   ],
  6.   "active": "LaW1urRQ"
  7. }

Note that in this example, the value LaW1urRQ is the ID of the key, not the key material itself.

Export the Keyring

Before going further, export the keyring. The exported material can be re-imported to the cluster in the event of an outage, or to restore a previously-deleted key:

  1. $ curl -XPOST -s localhost:8001/keyring/export | jq
  2. {
  3.   "data": "eyJrIjoiV1JZeTdubDlYeFZpR3VVQWtWTXBcL0JiVW1jMWZrWHluc0dKd3N4M1c0MlIxWE5XM05lZ05sdFdIVmJ1d0ZnaVZSTnFSdmM1WERscGY3b0NIZ1ZDQ3JvTFJ4czFnRURhOXpJT0tVV0prM2lhd0VLMHpKTXdwRDd5ZjV2VFYzQTY0Y2UxcVl1emJoSTI4VUZ1ZExRZWljVjd2T3BYblVvU3dOY3IzblhJQWhyWlcxc1grWXE3aHM1RzhLRXY2OWlRamJBTXAwbHZmTWNFWWxTOW9NUjdnSm5xZWlST0J1Q09iMm5tSXg0Qk1uaTJGalZzQzBtd2R2dmJyYWxYa3VLYXhpRWZvQm9EODk3MEtVcDYzY05lWGdJclpjang4YmJDV1lDRHlEVmExdGt5c0g1TjBJM0hTNDRQK1dyT2JkcElCUk5vSVZVNis1QWdcLzdZM290RUdzN1E9PSIsImQiOiIxWEZJOXZKQ05CTW5uVTB5c0hQenVjSG5nc2c5UURxQmcxZ3g1VVYxNWNlOEVTTlZXTmthYm8zdlUzS2VRTURcL0RUYXdzZCtJWHB5SllBTkRtanZNcytqU2lrVTFiRkpyMEVcLzBSRlg2emJrT0oybTR2bXlxdVE9PSIsIm4iOiJUUmRLK01Qajh6MkdHTmtyIn0="
  4. }

The response generated is an opaque blob containing the keyring, encrypted with a randomly-generated symmetric key. This random key is encrypted with the public RSA key defined via the keyring_public_key Kong configuration value.

The exported keyring should be stored in a safe location for disaster recovery purposes. It is not designed to be modified or decrypted before being used during a disaster recovery process.

Exercise the Encryption Routines

Create a Consumer with a basic-auth credential. At this point, the password field of the basic-auth credential will be symmetrically encrypted before it is written to the database (in addition to being hashed by the basic-auth plugin, which is done by the plugin regardless of whether keyring encryption is enabled):

  1. $ curl -s localhost:8001/consumers -d username=bob | jq
  2. {
  3.   "custom_id": null,
  4.   "created_at": 1576518610,
  5.   "id": "6375b5fd-9c95-4822-b2dd-80ffbccb7ec9",
  6.   "tags": null,
  7.   "username": "bob",
  8.   "type": 0
  9. }
  1. $ curl -s localhost:8001/consumers/bob/basic-auth -d username=bob -d password=supersecretpassword | jq
  2. {
  3.   "created_at": 1576518704,
  4.   "consumer": {
  5.     "id": "6375b5fd-9c95-4822-b2dd-80ffbccb7ec9"
  6.   },
  7.   "id": "fc46ce48-c1d6-4078-9f51-5a777350a8a2",
  8.   "password": "da61c0083b6d19ef3db2490d0da96a71572da0fa",
  9.   "username": "bob"
  10. }

Note that the returned password field in the API response is not the encrypted value; database encryption does not modify responses generated by the Admin API. This allows existing workflows to continue in the same form, while providing encryption at rest security under the hood. We can verify this by examining the value stored in the database:

  1. kong=# select id,password from basicauth_credentials;
  2.                   id                  |                                                         password
  3. --------------------------------------+---------------------------------------------------------------------------------------------------------------------------
  4.  fc46ce48-c1d6-4078-9f51-5a777350a8a2 | $ke$1$-LaW1urRQ-0f5b1ee8ddeefca1a1d75125-53f158a5f619133a2113692a7057e2b91fa947321de4480d452dd42c36bc9ef8aa6499cd429db6d7
  5. (1 row)

We can also verify that reading back the credential after it has been created behaves as expected:

  1. $ curl -s localhost:8001/consumers/bob/basic-auth/fc46ce48-c1d6-4078-9f51-5a777350a8a2 | jq
  2. {
  3.   "created_at": 1576518704,
  4.   "consumer": {
  5.     "id": "6375b5fd-9c95-4822-b2dd-80ffbccb7ec9"
  6.   },
  7.   "id": "fc46ce48-c1d6-4078-9f51-5a777350a8a2",
  8.   "password": "da61c0083b6d19ef3db2490d0da96a71572da0fa",
  9.   "username": "bob"
  10. }

Exercise Importing the Keyring

Restart Kong and re-import the previously exported keyring:

  1. $ kong restart
  1. $ curl localhost:8001/keyring/import -d data=<exported data>

This operation requires that the keyring_private_key point to the private RSA key associated with the public key used during the initial keyring export. Once this is complete, Admin API operations that require the keyring for encryption/ decryption can be verified:

  1. $ curl -s localhost:8001/consumers/bob/basic-auth/fc46ce48-c1d6-4078-9f51-5a777350a8a2 | jq
  2. {
  3.   "created_at": 1576518704,
  4.   "consumer": {
  5.     "id": "6375b5fd-9c95-4822-b2dd-80ffbccb7ec9"
  6.   },
  7.   "id": "fc46ce48-c1d6-4078-9f51-5a777350a8a2",
  8.   "password": "da61c0083b6d19ef3db2490d0da96a71572da0fa",
  9.   "username": "bob"
  10. }

Rotating Key Material

As noted above, Kong supports rotating keys by allowing for multiple keys to exist on the keyring at the same time. This allows for data fields written by one key to be read back, while a fresher encryption key is used for write operations. Rotating keys is a matter of importing or generating a new key into the keyring, and marking it as active. Arbitrary key material can be imported via the /keyring/import material, or Kong can generate a new key via /keyring/generate endpoint:

  1. $ curl -XPOST -s localhost:8001/keyring/generate
  2. {
  3.   "key": "t6NWgbj3g9cbNVC3/D6oZ2Md1Br5gWtRrqb1T2FZy44=",
  4.   "id": "8zgITLQh"
  5. }

Note that as a convenience the raw key material is returned from this endpoint call.

Once a new key is present in the keyring, activate the key’s ID:

  1. $ curl -s localhost:8001/keyring/activate -d key=8zgITLQh

Kong can write new sensitive data fields with the current active key, and read previously written fields in the database with the prior key, provided that key is in the keyring. Kong automatically selects the appropriate key to use when decrypting fields from the database.

At this point, it is encouraged to take another backup of the keyring via the /keyring/export Admin API endpoint.

Rotating Encrypted Data Fields

Once the keyring has updated, existing encrypted fields can rotate to use the new active key. To accomplish this rotation, issue a PATCH request to the entity in question:

  1. $ # curl -XPATCH -s localhost:8001/consumers/bob/basic-auth/fc46ce48-c1d6-4078-9f51-5a777350a8a2 -d password=adifferentsecretpassword | jq
  2. {
  3.   "created_at": 1576518704,
  4.   "consumer": {
  5.     "id": "6375b5fd-9c95-4822-b2dd-80ffbccb7ec9"
  6.   },
  7.   "id": "fc46ce48-c1d6-4078-9f51-5a777350a8a2",
  8.   "password": "cc7274e94e41f3e00c238ff8712d1a83693f2a89",
  9.   "username": "bob"
  10. }

Again, note that the encryption behavior is transparent to the Admin API response. Under the hood, Kong reads the entity’s password field with a read-only encryption key. As part of the PATCH process, Kong writes the password field back to the database with the new active key. To verify this process, examine the raw contents of the database:

  1. kong=# select id,password from basicauth_credentials;
  2.                   id                  |                                                         password
  3. --------------------------------------+---------------------------------------------------------------------------------------------------------------------------
  4.  fc46ce48-c1d6-4078-9f51-5a777350a8a2 | $ke$1$-8zgITLQh-b8d28531252241e6b95907e4-0768a9a4baaa2c777d9406d4e3098d813be55ae82c4c849182b06b9c5954704c6290c9e677bcd693
  5. (1 row)

Note that the key identifier within the password data blob has been updated. This rotation mechanism allows organizations to meet specific compliance needs around key management rotation and retention policies.

Currently, encrypted fields must undergo a direct write operation in order to rotate the encrypted field. Future releases of Kong may contain helper API operations to automate this process.

Exploring Missing Keyring Material Behavior

As a test, stop all Kong nodes in the cluster, and restart one Kong node again, but do not import the keyring material. The behavior of attempting to read an entity with an encrypted field now changes:

  1. $ time curl -s localhost:8001/consumers/bob/basic-auth/fc46ce48-c1d6-4078-9f51-5a777350a8a2
  2. {"message":"An unexpected error occurred"}
  3. real    0m24.811s
  4. user    0m0.017s
  5. sys    0m0.006s

When the Kong node restarts, it sends a broadcast request for the keyring, but as no other nodes are present in the cluster (or no other nodes had the keyring available), Kong is unable to decrypt the password field on the credential. Because Kong has an eventually-consistent clustering model, it makes several attempts to request the keyring and allows for delays in hearing responses from another cluster node; thus, the request takes several seconds to complete before finally failing.

Implementation Details

Encryption/Decryption

Kong Gateway uses 256-bit AES encryption in GCM mode. Cryptographic nonce values for each encryption routine execution are derived from the kernel CSPRNG (/dev/urandom). The AES routines used by Kong are provided by the OpenSSL library bundled with the Kong Gateway package.

Key Generation and Lifecycle

Kong Gateway’s keyring handling mechanisms allow for more than one key to be present on any given Kong node at a time. Each key may be used to read encrypted fields from the database, but one and only one key at any given time is used to write encrypted fields back to the data store. This process allows for a key rotation mechanism wherein new keyring material is introduced, and older keys may be present for a time to allow rotating previously-encrypted fields.

Through the kernel CSPRNG, Kong derives keyring material generated through the /keyring/generate Admin API endpoint. Kong stores keyring material in a shared memory zone that all Kong worker processes access. To prevent key material from being written to disk as part of memory paging operations, we recommend that swap be disabled on systems running Kong

When operating in cluster mode, keyring material propagates automatically among all nodes in the Kong cluster. Because Kong nodes do not have a notion of direct peer-to-peer communication, the underlying datastore serves as a communication channel to transmit messages. When a Kong node starts, it generates an ephemeral RSA key pair. The node’s public keys propagate to all other active nodes in the cluster. When an active node sees a message request for keyring material, it wraps the in-memory keyring material in the presented public key, and transmits the payload back over the central messaging channel provided by the underlying data store. This process allows each node in the cluster to broadcast keyring material to new nodes, without sending key material in plaintext over the wire. This model requires that at least one node be running at all times within the cluster; a failure of all nodes requires manually re-importing the keyring to one node during an outage recovery.

Encrypted Fields

The keyring module encrypts the following fields at rest:

  • key fields of certificate objects (corresponding to the private key of a TLS certificate).
  • Certain configuration parameters in plugins and plugin-related authentication objects. For information on which plugin fields are encrypted, see each individual plugin’s documentation.

Vault Integration

Kong’s keyring mechanism can integrate directly with HashiCorp Vault for keyring storage and versioning. In this model, Kong nodes read keyring material directly from a Vault KV secrets engine, rather than generating and disseminating keyring material around the cluster.

To configure Kong to use Vault for keyring storage, set the keyring_strategy configuration value to vault. Leveraging Vault also requires defining a host, mount point, and token for Vault access. See the Kong configuration reference for more details.

Key Format

Kong leverages version 2 of the Vault KV secrets engine. This process allows for the same versioning and key rotation mechanisms that the cluster keyring strategy provides. Each version of a KV secrets entry must contain both an id field, and a key field, e.g.:

  1. {
  2.   "key": "t6NWgbj3g9cbNVC3/D6oZ2Md1Br5gWtRrqb1T2FZy44=",
  3.   "id": "8zgITLQh"
  4. }

To provide consistent consumption of all Vault KV secrets, the underlying symmetric key is derived as the SHA256 of the key component of the secret value. Take note of this derivation when extending or re-using symmetric encryption keys from other systems. This derivation also implies that the key field can contain any arbitrary data, as Kong hashes the contents before importing the material into the keyring.

To provide a new key, add a new version to the Vault secrets engine at the configured path. The current_version of the Vault secret denotes the active key in the keyring. See the KV v2 documentation for more detail.

Vault Permissions

In order to communicate with Vault, Kong must be provided a Vault token for access. The token must be associated with a policy that allows the read and list actions on the path where keyring secrets are stored. Kong does not write keyring material to the Vault cluster.

Syncing the Keyring

Kong reads the keyring material from Vault when the Kong process starts. Any changes to the Vault KV store are not reflected on the Kong node until Kong syncs with Vault via the /keyring/vault/sync Admin API endpoint. This allows Kong to receive a Vault token with a low TTL, as the list and read operation only occur once.

Keyring on hybrid mode

Because Keyring encrypts the data in the database, it means it doesn’t encrypt data on Kong data plane nodes that run without a database and get data from the control plane.