Vault as the Secrets Backend - Systems Integration

Vault as the Secrets Backend - Systems Integration

Overview

At a high level, configuring a systems integration of Vault with Consul on Kubernetes consists of 1) a one time setup on Vault and 2) a setup of the secrets backend per Consul datacenter via Helm.

One time setup on Vault

Enabling Vault KV Secrets Engine - Version 2 to store arbitrary secrets
Enabling Vault PKI Engine if you are choosing to store and manage either Consul Server TLS credentials or Service Mesh and Consul client TLS credentials

Setup per Consul datacenter

Installing the Vault Injector within the Consul datacenter installation
Configuring a Kubernetes Auth Method in Vault to authenticate and authorize operations from the Consul datacenter
Enable Vault as the Secrets Backend in the Consul datacenter

One time setup on Vault

A one time setup on a Vault deployment is necessary to enable both the Vault KV Secrets Engine and the Vault PKI Engine. These docs assume that you have already setup a Vault cluster for use with Consul on Kubernetes.

Please read Run Vault on Kubernetes if instructions on setting up a Vault cluster are needed.

Vault KV Secrets Engine - Version 2

The following secrets can be stored in Vault KV secrets engine, which is meant to handle arbitrary secrets:

ACL Bootstrap token (global.acls.bootstrapToken)
ACL Partition token (global.acls.partitionToken)
ACL Replication token (global.acls.replicationToken)
Gossip encryption key (global.gossipEncryption)
Enterprise license (global.enterpriseLicense)
Snapshot Agent config (client.snapshotAgent.configSecret)

In order to store any of these secrets, we must enable the Vault KV secrets engine - Version 2.

$ vault secrets enable -path=consul kv-v2

$ vault secrets enable -path=consul kv-v2

Vault PKI Engine

The Vault PKI Engine must be enabled in order to leverage Vault for issuing Consul Server TLS certificates. More details for configuring the PKI Engine is found in Bootstrapping the PKI Engine under the Server TLS section.

$ vault secrets enable pki

$ vault secrets enable pki

Setup per Consul datacenter

After configuring Vault, Consul datacenters on Kubernetes must be deployed with the Vault Agent injector and configured to leverage the Vault Kubernetes Auth Method to read secrets from a Vault cluster.

Set Environment Variables to ensure integration consistency

Before installing the Vault Injector and configuring the Vault Kubernetes Auth Method, some environment variables need to be set to better ensure consistent mapping between Vault and Consul on Kubernetes.

DATACENTER

Recommended value: value of global.datacenter in your Consul Helm values file.
```
$ export DATACENTER=dc1
```
```
$ export DATACENTER=dc1
```

VAULT_AUTH_METHOD_NAME

Recommended value: a concatenation of a kubernetes- prefix (to denote the auth method type) with DATACENTER environment variable.
```
$ export VAULT_AUTH_METHOD_NAME=kubernetes-${DATACENTER}
```
```
$ export VAULT_AUTH_METHOD_NAME=kubernetes-${DATACENTER}
```

VAULT_SERVER_HOST

Recommended value: find the external IP address of your Vault cluster.

If Vault is installed in a Kubernetes cluster, get the external IP or DNS name of the Vault server load balancer.

On GKE or AKS, it’ll be an IP:

$ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

$ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

On EKS, it’ll be a hostname:

$ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

$ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

If Vault is not running on Kubernetes, utilize the api_addr as defined in the Vault High Availability Parameters configuration:

$ export VAULT_SERVER_HOST=<external IP for vault cluster>

$ export VAULT_SERVER_HOST=<external IP for vault cluster>

VAULT_ADDR

Recommended value: Connecting to port 8200 of the Vault server
```
$ export VAULT_ADDR=http://${VAULT_SERVER_HOST}:8200
```
```
$ export VAULT_ADDR=http://${VAULT_SERVER_HOST}:8200
```
Note: If your vault installation is current exposed using SSL, this address will need to use https instead of http. You will also need to setup the VAULT_CACERT environment variable.

VAULT_TOKEN

Recommended value: Your allocated Vault token. If running Vault in dev mode, this can be set to to root.
```
$ export VAULT_ADDR=<vault token>
```
```
$ export VAULT_ADDR=<vault token>
```

Install Vault Injector in your Consul k8s cluster

A minimal valid installation of Vault Kubernetes must include the Agent Injector which is utilized for accessing secrets from Vault. Vault servers could be deployed external to Vault on Kubernetes with the injector.externalVaultAddr value in the Vault Helm Configuration.

$ cat <<EOF >> vault-injector.yaml
# vault-injector.yaml
server:
  enabled: false
injector:
  enabled: true
  externalVaultAddr: ${VAULT_ADDR}
  authPath: auth/${VAULT_AUTH_METHOD_NAME}
EOF

$ cat <<EOF >> vault-injector.yaml
# vault-injector.yaml
server:
  enabled: false
injector:
  enabled: true
  externalVaultAddr: ${VAULT_ADDR}
  authPath: auth/${VAULT_AUTH_METHOD_NAME}
EOF

Issue the Helm install command to install the Vault agent injector using the HashiCorp Vault Helm chart.

$ helm install vault-${DATACENTER} -f vault-injector.yaml hashicorp/vault --wait

$ helm install vault-${DATACENTER} -f vault-injector.yaml hashicorp/vault --wait

Configure the Kubernetes Auth Method in Vault for the datacenter

Enable the Auth Method

Ensure that the Vault Kubernetes Auth method is enabled.

$ vault auth enable -path=kubernetes-${DATACENTER} kubernetes

$ vault auth enable -path=kubernetes-${DATACENTER} kubernetes

Configure Auth Method with JWT token of service account

After enabling the Kubernetes auth method, in Vault, ensure that you have configured the Kubernetes Auth method properly as described in Kubernetes Auth Method Configuration.

First, while targeting your Consul cluster, get the externally reachable address of the Consul Kubernetes cluster.

$ export KUBE_API_URL=$(kubectl config view -o jsonpath="{.clusters[?(@.name == \"$(kubectl config current-context)\")].cluster.server}")

$ export KUBE_API_URL=$(kubectl config view -o jsonpath="{.clusters[?(@.name == \"$(kubectl config current-context)\")].cluster.server}")

Next, you will configure the Vault Kubernetes Auth Method for the datacenter. You will need to provide it with:

token_reviewer_jwt - this a JWT token from the Consul datacenter cluster that the Vault Kubernetes Auth Method will use to query the Consul datacenter Kubernetes API when services in the Consul datacenter request data from Vault.
kubernetes_host - this is the URL of the Consul datacenter’s Kubernetes API that Vault will query to authenticate the service account of an incoming request from a Consul data center kubernetes service.
kubernetes_ca_cert - this is the CA certification that is currently being used by the Consul datacenter Kubernetes cluster.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://${KUBE_API_URL}:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://${KUBE_API_URL}:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Enable Vault as the Secrets Backend in the Consul datacenter

Finally, you will configure the Consul on Kubernetes helm chart for the datacenter to expect to receive the following values (if you have configured them) to be retrieved from Vault:

ACL Bootstrap token (global.acls.bootstrapToken)
ACL Partition token (global.acls.partitionToken)
ACL Replication token (global.acls.replicationToken)
Enterprise license (global.enterpriseLicense)
Gossip encryption key (global.gossipEncryption)
Snapshot Agent config (client.snapshotAgent.configSecret)
TLS CA certificates (global.tls.caCert)
Server TLS certificates (server.serverCert)

global:
  secretsBackend:
    vault:
      enabled: true

values.yaml

global:
  secretsBackend:
    vault:
      enabled: true

Next Steps

As a next step, please proceed to Vault integration with Consul on Kubernetes’ Data Integration.

Troubleshooting

The Vault integration with Consul on Kubernetes makes use of the Vault Agent Injectors. Kubernetes annotations are added to the deployments of the Consul components which cause the Vault Agent Injector to be added as an init-container that will then attach Vault secrets to Consul’s pods at startup. Additionally the Vault Agent sidecar is added to the Consul component pods which is responsible for synchronizing and reissuing secrets at runtime. As a result of these additional sidecar containers the typical location for logging is expanded in the Consul components.

As a general rule the best way to troubleshoot startup issues for your Consul installation when using the Vault integration is to establish if the vault-agent-init container has completed or not via kubectl logs -f <your-consul-component> -c vault-agent-int and checking to see if the secrets have completed rendering.

If the secrets are not properly rendered the underlying problem will be logged in vault-agent-init init-container and generally is related to the Vault Kube Auth Role not having the correct policies for the specific secret e.g. global.secretsBackend.vault.consulServerRole not having the correct policies for TLS.
If the secrets are rendered and the vault-agent-init container has completed AND the Consul component has not become Ready, this generally points to an issue with Consul being unable to utilize the Vault secret. This can occur if, for example, the Vault Role created for the PKI engine does not have the correct alt_names or otherwise is not properly configured. The best logs for this circumstance are the Consul container logs: kubectl logs -f <your-consul-component> -c consul.