Manual Secure Configuration of Consul on AWS Elastic Container Service (ECS)

This topic describes how to enable Consul security features for your production workloads.

Prerequisites

The following features must be configured for your Consul server cluster:

• TLS encryption for RPC communication between Consul clients and servers.
• Gossip encryption for encrypting gossip traffic.
• Access control lists (ACLs) for authentication and authorization for Consul clients and services on the mesh.

You should already have followed the manual installation instructions to define the necessary components of the task definition for Consul on ECS.

You should be familiar with specifying sensitive data on ECS.

You should be familiar with configuring Consul’s secure features, including how to create ACL tokens and policies. Refer to the ACL system documentation and Day 1: Security tutorials for an introduction and additional information.

Auth Method

Tokens are artifacts within the ACL system that authenticate users, services, and Consul agents. Tokens are linked to policies that specify the resources the token bearer has access to when making requests in the network.

Auth Methods are a Consul server component that performs authentication against a trusted external party to authorize the creation of ACL tokens. The AWS IAM auth method is used to enable an ECS task to automatically obtain ACL tokens when the task starts up.

There are two types of ACL tokens for Consul on ECS:

• Client tokens: used by the consul-client containers to join the Consul cluster
• Service tokens: used by sidecar containers for service registration and health syncing

This section describes how to manually configure the AWS IAM auth method for Consul on ECS. Alternatively, you can install the ACL controller to ease the burden of creating these resources. The ACL controller can automatically configure ACL resources for Consul on ECS. For additional details, refer to ACL Controller and Architecture.

ECS Task Role Configuration

The ECS task role is an IAM role associated with an ECS task.

When an ECS task starts up, it runs a consul login command. The consul login command obtains credentials for the task role role from AWS. It uses those credentials to sign the login request to the AWS IAM auth method. This proves the ECS task’s identity to the Consul servers.

The task role must be configured with the following details to compatible with the AWS IAM auth method.

• An iam:GetRole permission to fetch itself. See IAM Policies for details.
• A consul.hashicorp.com.service-name tag on the task role which contains the Consul service name for the application in this task.
• EnterpriseEnterprise A consul.hashicorp.com.namespace tag on the task role indicating the Consul Enterprise namespace where this service is registering.

The following sections describe how to configure the auth method to enable task roles to successfully authenticate to the AWS IAM auth method and obtain tokens with the necessary permissions.

Auth Method for Client Tokens

The following steps configure an instance of the auth method that creates client tokens for tasks.

1. Create the auth method instance
2. Create the client policy and role
3. Create the binding rule

Create Auth Method for Client Tokens

The following Consul CLI command creates an instance of the auth method for client tokens.

consul acl auth-method create \
    -type aws-iam \
    -name iam-ecs-client-token \
    -description="AWS IAM auth method for ECS client tokens" \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true
}'

The following flags are required:

In the -config option, the following fields are required:

Create Client Policy and Role

Configure the following ACL policy for Consul client tokens:

client-token-policy.hcl

node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}

The policy allows node:write for any node name, which is necessary because the Consul node names on ECS are not known until runtime.

You can add the policy in Consul using the consul acl policy create command or the [PUT] /v1/acl/policy API endpoint.

After the policy is created, create a Consul role associated with the policy by using the consul acl role create command or the [PUT] /v1/acl/role API endpoint.

The following example shows how to use the Consul CLI to create the client policy and role.

consul acl policy create -name consul-ecs-client-policy -rules @client-token-policy.hcl
consul acl role create -name consul-ecs-client-role -policy-name consul-ecs-client-policy

Create Binding Rule for Client Tokens

The following creates a binding rule for the auth method for client tokens. The binding rule associates the client role with each token created by a successful login to this auth method instance.

consul acl binding-rule create -method iam-ecs-client-token \
   -description 'Bind a role for Consul clients on ECS' \
   -bind-type role \
   -bind-name consul-ecs-client-role

Auth Method for Service Tokens

The following steps configure an instance of the auth method that creates service tokens for tasks.

• Create the auth method instance
• Create the binding rule

Create Auth Method for Service Tokens

The following uses the Consul CLI to create an instance of the auth method for service tokens. This configures the auth method to associate a service identity to each token created during login to this auth method instance.

consul acl auth-method create \
    -type aws-iam \
    -name iam-ecs-service-token \
    -description="AWS IAM auth method for ECS service tokens" \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true,
  "IAMEntityTags": [
    "consul.hashicorp.com.service-name"
  ]
}'

The following flags are required:

In the -config option, the following fields are required:

The following binding rule is used to associate a service identity with each token created by successful login to this instance of the auth method. The service identity name is taken from the consul.hashicorp.com.service-name tag from the authenticating IAM role identity.

Create Binding Rule

consul acl binding-rule create \
   -method iam-ecs-service-token \
   -description 'Bind a service identity from IAM role tags for ECS service tokens' \
   -bind-type service \
   -bind-name '${entity_tags.consul.hashicorp.com.service-name}'

Configuration for Consul EnterpriseEnterpriseEnterprise

When using Consul Enterprise namespaces and admin partitions, pass the -partition <partition-name> option to the Consul CLI when creating Consul ACL roles, policies, auth methods, and binding rules, in order to create these resources in a particular partition.

The following shows how to create the ACL policy for client tokens. This ensures permissions for the client token are scoped to a particular partition.

client-token-policy-ent.hcl

partition "<partition-name>" {
  node_prefix "" {
    policy = "write"
  }
  namespace_prefix "" {
    service_prefix "" {
      policy = "read"
    }
  }
}

The following commands show how to use the Consul CLI to create the policy and role.

consul acl policy create -partition <partition-name> \
    -name consul-ecs-client-policy \
    -rules @client-token-policy-ent.hcl
consul acl role create \
  -partition <partition-name> \
  -name consul-ecs-client-role \
  -policy-name consul-ecs-client-policy

The auth method for service tokens requires the following additional configuration to include a namespace binding rule. This ensures the service tokens are created in the right namespace during login. (The namespace binding rule must not be configured on the instance of the auth method instance for client tokens.)

consul acl auth-method create \
    -partition <partition-name> \
    -type aws-iam \
    -name iam-ecs-service-token \
    -description="AWS IAM auth method for ECS service tokens" \
    -namespace-rule-selector 'entity_tags["consul.hashicorp.com.namespace"] != ""' \
    -namespace-rule-bind-namespace '${entity_tags.consul.hashicorp.com.namespace}' \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true,
  "IAMEntityTags": [
    "consul.hashicorp.com.service-name",
    "consul.hashicorp.com.namespace"
  ]
}'

Secret storage

You should securely store the following secrets in order to make them available to ECS tasks.

1. Consul Server CA certificates. More than one may be required for different Consul protocols.
2. Consul gossip encryption key

These secrets can be securely stored and passed to ECS tasks using either of the following AWS secret services:

• AWS Systems Manager Parameter Store
• AWS Secrets Manager

Once the secrets are stored they can be referenced using their ARN. The following shows example secret ARNs when using AWS Secrets Manager:

Configure consul-client

The following secrets must be passed to the consul-client container:

• Consul server CA certificates
• Gossip encryption key

The following example shows how to include these secrets in the task definition. The secrets list specifies environment variable names that will be set to the secret values for this container. ECS automatically fetches the secret values specified in the valueFrom fields during task provisioning.

{
  "containerDefinitions": [
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      "secrets": [
        {
          "name": "CONSUL_CACERT_PEM",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert"
        },
        {
          "name": "CONSUL_HTTPS_CACERT_PEM",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-https-ca-cert"
        },
        {
          "name": "CONSUL_GOSSIP_ENCRYPTION_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key"
        }
      ]
    },
    ...
  ]
}

Next, update the Consul client startup script with the consul login command and additional Consul configuration options for a secure configuration.

The following is an example of the additional content to include in the consul-client startup script. Refer to the install page for the remainder of the startup script and how to pass this script to the container.

...
# Obtain details from the task metadata
ECS_TASK_META=$(curl -s $ECS_CONTAINER_METADATA_URI_V4/task)
TASK_REGION=$(echo "$ECS_TASK_META" | jq --raw-output '.TaskARN / ":" | .[3]')
TASK_ID=$(echo "$ECS_TASK_META" | jq --raw-output '.TaskARN / "/" | .[2]')
CLUSTER_ARN=$(echo "$ECS_TASK_META" | jq --raw-output '.TaskARN | sub(":task/(?<cluster>[^/]+).*"; ":cluster/\(.cluster)")')
# Write the CA certs to a files in the shared volume
echo "$CONSUL_CACERT_PEM" > /consul/consul-ca-cert.pem
echo "$CONSUL_HTTPS_CACERT_PEM" > /consul/consul-https-ca-cert.pem
consul_login() {
    echo "Logging into auth method"
    consul login \
      -http-addr "<consul server address>" \
      -ca-file /consul/consul-https-ca-cert.pem \
      -partition "<partition>" \
      -type aws-iam \
      -method iam-ecs-client-token \
      -meta "consul.hashicorp.com/task-id=$TASK_ID" \
      -meta "consul.hashicorp.com/cluster=$CLUSTER_ARN" \
      -aws-region "$TASK_REGION" \
      -aws-auto-bearer-token -aws-include-entity \
      -token-sink-file /consul/client-token
}
read_token_stale() {
    consul acl token read -http-addr ${ consul_http_addr } \
      -ca-file /consul/consul-https-ca-cert.pem \
      -stale -self -token-file /consul/client-token \
      > /dev/null
}
# Retry in order to login successfully.
while ! consul_login; do
    sleep 2
done
# Allow the health-sync container to read this token for consul logout.
chmod 0644 /consul/client-token
# Wait for raft replication to hopefully occur. Without this, an "ACL not found" may be cached for a while.
COUNT=20
while [ "$COUNT" -gt 0 ]; do
    echo "Checking that the ACL token exists when reading it in the stale consistency mode ($COUNT attempts remaining)"
    if read_token_stale; then
        echo "Successfully read ACL token from the server"
        break
    fi
    sleep 0.1
    COUNT=$((COUNT - 1))
done
if [ "$COUNT" -eq 0 ]; then
   echo "Unable to read ACL token from a Consul server; please check that your server cluster is healthy"
   exit 1
fi
# This is interpolated into the agent-defaults.hcl
export AGENT_TOKEN=$(cat /consul/client-token)
# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
...
# Configure gossip encryption key
encrypt = "$CONSUL_GOSSIP_ENCRYPTION_KEY"
# Configure TLS settings
auto_encrypt = {
  tls = true
  ip_san = ["$ECS_IPV4"]
}
tls {
  defaults {
    ca_file = "/consul/consul-ca-cert.pem"
    verify_outgoing = true
  }
}
# Configure ACLs
acl {
  enabled = true
  default_policy = "deny"
  down_policy = "async-cache"
  tokens {
    agent = "$AGENT_TOKEN"
  }
}
partition = "<partition>"
EOF

The following describes the additional steps added to the startup script:

• Fetch additional details from the task metadata: the AWS region, task id, and cluster arn. These details are necessary for the consul login command used to obtain a Consul client token.
• Write CA certificates to files for Consul CLI and Consul client
• Run the consul login command in a retry loop.
• Wait for Raft replication to hopefully occur for this token.
• Configure the Consul client config file with additional fields necessary for secure configuration.

The following flags are passed to the consul login command:

Field name	Type	Description
-http-addr	string	HTTP(S) address of the Consul server.
-ca-file	string	Path of the CA cert for Consul’s HTTPS interface. Not required when using Consul servers on HCP.
-partition	string	EnterpriseEnterprise The Consul Enterprise admin partition the auth method belongs to.
-type	string	The auth method type. Must be aws-iam.
-method	string	The auth method name. Must be the name of the auth method for ECS client tokens.
-meta	string	Metadata to set in description of the created token. Should include the task id and cluster as shown.
-aws-region	string	The AWS region where the task is running.
-aws-auto-bearer-token		Must be set to login to the AWS IAM auth method.
-aws-include-entity		Must be set to enable IAM role details.

The following table describes the additional fields that must be included in the Consul client configuration file.

Field name	Type	Description
encrypt	string	Gossip encryption key
tls.defaults.ca_file	string	Consul server CA cert for TLS verification.
acl.enabled	boolean	Enable ACLs for this agent.
acl.tokens.agent	string	Consul client token which authorizes this agent with Consul servers.
partition	string	EnterpriseEnterprise The Consul Enterprise admin partition this agent belongs to.

Configure Audit Logging EnterpriseEnterprise

Audit logging is supported on clients running Consul Enterprise with ACLs enabled. To enable audit logging, update the startup script to add an audit stanza to the Consul client configuration file.

The following example modifies the consul-client startup script to configure audit logs to be written to the stdout of the consul-client container.

...
# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
...
partition = "<partition>"
audit {
  enabled = true
  sink "stdout" {
    type   = "file"
    format = "json"
    path   = "/dev/stdout"
    delivery_guarantee = "best-effort"
  }
}
EOF

The following table describes the fields that must be included to configure audit logging.

Configure consul-ecs-mesh-init and consul-ecs-health-sync

The following additional options should be set in the CONSUL_ECS_CONFIG_JSON environment variable. When these options are specified, the consul-ecs mesh-init command will run the consul login command to obtain a service token from the Consul AWS IAM Auth method. The consul-ecs health-sync command is responsible for running a consul logout command for both the service and client tokens when the task shuts down.

{
  "consulHTTPAddr": "<Consul server address>",
  "consulCACertFile": "/consul/consul-https-ca-cert.pem",
  "consulLogin": {
    "enabled": true,
    "method": "iam-ecs-service-token",
    "extraLoginFlags": ["-partition", "<partition>"]
  },
  "bootstrapDir": "/consul",
  "healthSyncContainers": [],
  ...
}

The following table explains the additional fields for the CONSUL_ECS_CONFIG_JSON:

Field name	Type	Description
consulHTTPAddr	string	HTTP(S) address for the Consul server.
consulCACertFile	string	Path of the CA cert file for Consul’s HTTPS interface. Not required for Consul servers in HCP.
consulLogin.enabled	boolean	Must be set to true to log in to the auth method.
consulLogin.method	string	Must be set to the name of the auth method instance for service tokens.
consulLogin.extraLoginFlags	list	Additional flags passed to the consul login command. EnterpriseEnterprise This shows how to pass the Consul Enterprise admin partition to the consul login command.