Custom Resource Definitions

This topic describes how to manage Consul configuration entries via Kubernetes Custom Resources. Configuration entries provide cluster-wide defaults for the service mesh.

Requirements

  • consul-helm 0.28.0 or later
  • consul-k8s 0.22.0 or later
  • consul 1.8.4 or later; some configuration entries require a newer version of Consul

Supported Configuration Entries

You can specify the following values in the kind field. Click on a configuration entry to view its documentation:

Installation

Verify that the minimum version of the helm chart (0.28.0) is installed:

  1. $ helm search repo hashicorp/consul
  2. NAME                CHART VERSION   APP VERSION DESCRIPTION
  3. hashicorp/consul    0.28.0          1.9.1       Official HashiCorp Consul Chart
  1. $ helm search repo hashicorp/consul
  2. NAME                CHART VERSION   APP VERSION DESCRIPTION
  3. hashicorp/consul    0.28.0          1.9.1       Official HashiCorp Consul Chart

Update your helm repository cache if necessary:

  1. $ helm repo update
  2. Hang tight while we grab the latest from your chart repositories...
  3. ...Successfully got an update from the "hashicorp" chart repository
  4. Update Complete. ⎈Happy Helming!⎈
  1. $ helm repo update
  2. Hang tight while we grab the latest from your chart repositories...
  3. ...Successfully got an update from the "hashicorp" chart repository
  4. Update Complete. ⎈Happy Helming!⎈

Next, you must configure consul-helm via your values.yaml to install the custom resource definitions and enable the controller that acts on them:

  1. global:
  2.   name: consul
  4. controller:
  5.   enabled: true
  7. connectInject:
  8.   enabled: true

Overview - 图1

values.yaml

  1. global:
  2.   name: consul
  4. controller:
  5.   enabled: true
  7. connectInject:
  8.   enabled: true

Note that:

  1. controller.enabled: true installs the CRDs and enables the controller.
  2. Configuration entries are used to configure Consul service mesh so it’s also expected that connectInject will be enabled.

See Install with Helm Chart for further installation instructions.

Upgrading An Existing Cluster to CRDs

If you have an existing Consul cluster running on Kubernetes you may need to perform extra steps to migrate to CRDs. See Upgrade An Existing Cluster to CRDs for full instructions.

Usage

Once installed, you can use kubectl to create and manage Consul’s configuration entries.

Create

You can create configuration entries via kubectl apply.

  1. $ cat <<EOF | kubectl apply --filename -
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceDefaults
  4. metadata:
  5.   name: foo
  6. spec:
  7.   protocol: "http"
  8. EOF
  10. servicedefaults.consul.hashicorp.com/foo created
  1. $ cat <<EOF | kubectl apply --filename -
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceDefaults
  4. metadata:
  5.   name: foo
  6. spec:
  7.   protocol: "http"
  8. EOF
  10. servicedefaults.consul.hashicorp.com/foo created

See Configuration Entries for detailed schema documentation.

Get

You can use kubectl get [kind] [name] to get the status of the configuration entry:

  1. $ kubectl get servicedefaults foo
  2. NAME  SYNCED
  3. foo   True
  1. $ kubectl get servicedefaults foo
  2. NAME  SYNCED
  3. foo   True

The SYNCED status shows whether the configuration entry was successfully created in Consul.

Describe

You can use kubectl describe [kind] [name] to investigate the status of the configuration entry. If SYNCED is false, the status will contain the reason why.

  1. $ kubectl describe servicedefaults foo
  2. Status:
  3.   Conditions:
  4.     Last Transition Time:  2020-10-09T21:15:50Z
  5.     Status:                True
  6.     Type:                  Synced
  1. $ kubectl describe servicedefaults foo
  2. Status:
  3.   Conditions:
  4.     Last Transition Time:  2020-10-09T21:15:50Z
  5.     Status:                True
  6.     Type:                  Synced

Edit

You can use kubectl edit [kind] [name] to edit the configuration entry:

  1. $ kubectl edit servicedefaults foo
  2. # change protocol: http => protocol: tcp
  3. servicedefaults.consul.hashicorp.com/foo edited
  1. $ kubectl edit servicedefaults foo
  2. # change protocol: http => protocol: tcp
  3. servicedefaults.consul.hashicorp.com/foo edited

You can then use kubectl get to ensure the change was synced to Consul:

  1. $ kubectl get servicedefaults foo
  2. NAME  SYNCED
  3. foo   True
  1. $ kubectl get servicedefaults foo
  2. NAME  SYNCED
  3. foo   True

Delete

You can use kubectl delete [kind] [name] to delete the configuration entry:

  1. $ kubectl delete servicedefaults foo
  2. servicedefaults.consul.hashicorp.com "foo" deleted
  1. $ kubectl delete servicedefaults foo
  2. servicedefaults.consul.hashicorp.com "foo" deleted

You can then use kubectl get to ensure the configuration entry was deleted:

  1. $ kubectl get servicedefaults foo
  2. Error from server (NotFound): servicedefaults.consul.hashicorp.com "foo" not found
  1. $ kubectl get servicedefaults foo
  2. Error from server (NotFound): servicedefaults.consul.hashicorp.com "foo" not found

Delete Hanging

If running kubectl delete hangs without exiting, there may be a dependent configuration entry registered with Consul that prevents the target configuration entry from being deleted. For example, if you set the protocol of your service to http via ServiceDefaults and then create a ServiceSplitter, you won’t be able to delete the ServiceDefaults.

This is because by deleting the ServiceDefaults config, you are setting the protocol back to the default which is tcp. Since ServiceSplitter requires that the service has an http protocol, Consul will not allow the ServiceDefaults to be deleted since that would put Consul into a broken state.

In order to delete the ServiceDefaults config, you would need to first delete the ServiceSplitter.

Kubernetes Namespaces

Consul OSS

Consul Open Source (Consul OSS) ignores Kubernetes namespaces and registers all services into the same global Consul registry based on their names. For example, service web in Kubernetes namespace web-ns and service admin in Kubernetes namespace admin-ns will be registered into Consul as web and admin with the Kubernetes source namespace ignored.

When creating custom resources to configure these services, the namespace of the custom resource is also ignored. For example, you can create a ServiceDefaults custom resource for service web in the Kubernetes namespace admin-ns even though the web service is actually running in the web-ns namespace (although this is not recommended):

  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5.   namespace: admin-ns
  6. spec:
  7.   protocol: http
  8. ---
  9. apiVersion: apps/v1
  10. kind: Deployment
  11. metadata:
  12.   name: web
  13.   namespace: web-ns
  14. spec: ...
  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5.   namespace: admin-ns
  6. spec:
  7.   protocol: http
  8. ---
  9. apiVersion: apps/v1
  10. kind: Deployment
  11. metadata:
  12.   name: web
  13.   namespace: web-ns
  14. spec: ...

NOTE: If two custom resources of the same kind and the same name are attempted to be created in different Kubernetes namespaces, the last one created will not be synced.

ServiceIntentions Special Case

ServiceIntentions are different from the other custom resources because the name of the resource doesn’t matter. For other resources, the name of the resource determines which service it configures. For example, this resource configures the service web:

  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5. spec:
  6.   protocol: http
  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5. spec:
  6.   protocol: http

For ServiceIntentions, because we need to support the ability to create wildcard intentions (e.g. foo => * (allow) meaning that foo can talk to any service), and because * is not a valid Kubernetes resource name, we instead use the field spec.destination.name to configure the destination service for the intention:

  1. # foo => * (allow)
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceIntentions
  4. metadata:
  5.   name: name-does-not-matter
  6. spec:
  7.   destination:
  8.     name: '*'
  9.   sources:
  10.     - name: foo
  11.       action: allow
  12. ---
  13. # foo => web (allow)
  14. apiVersion: consul.hashicorp.com/v1alpha1
  15. kind: ServiceIntentions
  16. metadata:
  17.   name: name-does-not-matter
  18. spec:
  19.   destination:
  20.     name: web
  21.   sources:
  22.     - name: foo
  23.       action: allow
  1. # foo => * (allow)
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceIntentions
  4. metadata:
  5.   name: name-does-not-matter
  6. spec:
  7.   destination:
  8.     name: '*'
  9.   sources:
  10.     - name: foo
  11.       action: allow
  12. ---
  13. # foo => web (allow)
  14. apiVersion: consul.hashicorp.com/v1alpha1
  15. kind: ServiceIntentions
  16. metadata:
  17.   name: name-does-not-matter
  18. spec:
  19.   destination:
  20.     name: web
  21.   sources:
  22.     - name: foo
  23.       action: allow

NOTE: If two ServiceIntentions resources set the same spec.destination.name, the last one created will not be synced.

Consul Enterprise

Enterprise

Consul Enterprise supports multiple configurations for how Kubernetes namespaces are mapped to Consul namespaces. The Consul namespace that the custom resource is registered into depends on the configuration being used but in general, you should create your custom resources in the same Kubernetes namespace as the service they’re configuring and everything will work as expected.

The details on each configuration are:

  1. Mirroring - The Kubernetes namespace will be “mirrored” into Consul, i.e. service web in Kubernetes namespace web-ns will be registered as service web in the Consul namespace web-ns. In the same vein, a ServiceDefaults custom resource with name web in Kubernetes namespace web-ns will configure that same service.

    This is configured via connectInject.consulNamespaces:

    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     mirroringK8S: true
    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     mirroringK8S: true

  2. Mirroring with prefix - The Kubernetes namespace will be “mirrored” into Consul with a prefix added to the Consul namespace, i.e. if the prefix is k8s- then service web in Kubernetes namespace web-ns will be registered as service web in the Consul namespace k8s-web-ns. In the same vein, a ServiceDefaults custom resource with name web in Kubernetes namespace web-ns will configure that same service.

    This is configured via connectInject.consulNamespaces:

    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     mirroringK8S: true
    8.     mirroringK8SPrefix: k8s-
    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     mirroringK8S: true
    8.     mirroringK8SPrefix: k8s-

  3. Single destination namespace - The Kubernetes namespace is ignored and all services will be registered into the same Consul namespace, i.e. if the destination Consul namespace is my-ns then service web in Kubernetes namespace web-ns will be registered as service web in Consul namespace my-ns.

    In this configuration, the Kubernetes namespace of the custom resource is ignored. For example, a ServiceDefaults custom resource with the name web in Kubernetes namespace admin-ns will configure the service with name web even though that service is running in Kubernetes namespace web-ns because the ServiceDefaults resource ends up registered into the same Consul namespace my-ns.

    This is configured via connectInject.consulNamespaces:

    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     consulDestinationNamespace: 'my-ns'
    1. global:
    2.   name: consul
    3.   enableConsulNamespaces: true
    4.   image: hashicorp/consul-enterprise:<tag>-ent
    5. connectInject:
    6.   consulNamespaces:
    7.     consulDestinationNamespace: 'my-ns'

    NOTE: In this configuration, if two custom resources of the same kind and the same name are attempted to be created in two Kubernetes namespaces, the last one created will not be synced.

ServiceIntentions Special Case (Enterprise)

ServiceIntentions are different from the other custom resources because the name of the resource doesn’t matter. For other resources, the name of the resource determines which service it configures. For example, this resource configures the service web:

  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5. spec:
  6.   protocol: http
  1. apiVersion: consul.hashicorp.com/v1alpha1
  2. kind: ServiceDefaults
  3. metadata:
  4.   name: web
  5. spec:
  6.   protocol: http

For ServiceIntentions, because we need to support the ability to create wildcard intentions (e.g. foo => * (allow) meaning that foo can talk to any service), and because * is not a valid Kubernetes resource name, we instead use the field spec.destination.name to configure the destination service for the intention:

  1. # foo => * (allow)
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceIntentions
  4. metadata:
  5.   name: name-does-not-matter
  6. spec:
  7.   destination:
  8.     name: '*'
  9.   sources:
  10.     - name: foo
  11.       action: allow
  12. ---
  13. # foo => web (allow)
  14. apiVersion: consul.hashicorp.com/v1alpha1
  15. kind: ServiceIntentions
  16. metadata:
  17.   name: name-does-not-matter
  18. spec:
  19.   destination:
  20.     name: web
  21.   sources:
  22.     - name: foo
  23.       action: allow
  1. # foo => * (allow)
  2. apiVersion: consul.hashicorp.com/v1alpha1
  3. kind: ServiceIntentions
  4. metadata:
  5.   name: name-does-not-matter
  6. spec:
  7.   destination:
  8.     name: '*'
  9.   sources:
  10.     - name: foo
  11.       action: allow
  12. ---
  13. # foo => web (allow)
  14. apiVersion: consul.hashicorp.com/v1alpha1
  15. kind: ServiceIntentions
  16. metadata:
  17.   name: name-does-not-matter
  18. spec:
  19.   destination:
  20.     name: web
  21.   sources:
  22.     - name: foo
  23.       action: allow

In addition, we support the field spec.destination.namespace to configure the destination service’s Consul namespace. If spec.destination.namespace is empty, then the Consul namespace used will be the same as the other config entries as outlined above.