Cluster Peering on Kubernetes

Cluster Peering on Kubernetes

Cluster peering is currently in technical preview: Functionality associated with cluster peering is subject to change. You should never use the technical preview release in secure environments or production scenarios. Features in technical preview may have performance issues, scaling issues, and limited support.

To establish a cluster peering connection on Kubernetes, you need to enable the feature in the Helm chart and create custom resource definitions for each side of the peering.

The following Custom Resource Definitions (CRDs) are used to create and manage a peering connection:

PeeringAcceptor: Generates a peering token and accepts an incoming peering connection.
PeeringDialer: Uses a peering token to make an outbound peering connection with the cluster that generated the token.

Prerequisites

You must implement the following requirements to create and use cluster peering connections with Kubernetes:

Consul 1.13 Alpha 2 or later
At least two Kubernetes clusters
The Kubernetes clusters must be running in a flat network
The network must be running on Consul on Kubernetes v.0.45 or later

Helm chart configuration

To establish cluster peering through Kubernetes, deploy clusters with the following Helm values.

global:
  image: "hashicorp/consul:1.13.0-alpha2"
  peering:
    enabled: true
connectInject:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1

values.yaml

global:
  image: "hashicorp/consul:1.13.0-alpha2"
  peering:
    enabled: true
connectInject:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1

Install Consul on Kubernetes on each Kubernetes cluster by applying values.yaml using the Helm CLI.

$ export HELM_RELEASE_NAME=cluster-name

$ export HELM_RELEASE_NAME=cluster-name

$ helm install ${HELM_RELEASE_NAME} hashicorp/consul --version "0.45.0" --values values.yaml

$ helm install ${HELM_RELEASE_NAME} hashicorp/consul --version "0.45.0" --values values.yaml

Create a peering connection

To peer Kubernetes clusters running Consul, you need to create a peering token and share it with the other cluster.

In cluster-01, create the PeeringAcceptor custom resource.

apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringAcceptor
metadata:
  name: cluster-02 ## The name of the peer you want to connect to
spec:
  peer:
    secret:
      name: "peering-token"
      key: "data"
      backend: "kubernetes"

acceptor.yml

apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringAcceptor
metadata:
  name: cluster-02 ## The name of the peer you want to connect to
spec:
  peer:
    secret:
      name: "peering-token"
      key: "data"
      backend: "kubernetes"

Apply the PeeringAcceptor resource to the first cluster.

$ kubectl apply --filename acceptor.yml

$ kubectl apply --filename acceptor.yml

Save your peering token so that you can export it to the other cluster.

$ kubectl get secret peering-token --output yaml > peering-token.yml

$ kubectl get secret peering-token --output yaml > peering-token.yml

Apply the peering token to the second cluster.

$ kubectl apply --filename peering-token.yml

$ kubectl apply --filename peering-token.yml

In “cluster-02,” create the PeeringDialer custom resource.

apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringDialer
metadata:
  name: cluster-01 ## The name of the peer you want to connect to
spec:
  peer:
    secret:
      name: "peering-token"
      key: "data"
      backend: "kubernetes"

dialer.yml

apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringDialer
metadata:
  name: cluster-01 ## The name of the peer you want to connect to
spec:
  peer:
    secret:
      name: "peering-token"
      key: "data"
      backend: "kubernetes"

Apply the PeeringDialer resource to the second cluster.

$ kubectl apply --filename dialer.yml

$ kubectl apply --filename dialer.yml

Deploy and export cluster services

For the service in “cluster-02” that you want to export, add the following annotations to your service’s pods. This service is referred to as “backend-service” in the following steps.

##…
annotations:
  "consul.hashicorp.com/connect-inject": "true"
  "consul.hashicorp.com/transparent-proxy": "false"
##…

backend-service.yml

##…
annotations:
  "consul.hashicorp.com/connect-inject": "true"
  "consul.hashicorp.com/transparent-proxy": "false"
##…

In “cluster-02,” create an ExportedServices custom resource.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ExportedServices
metadata:
  name: default ## The name of the partition containing the service
spec:
  services:
    name: backend-service ## The name of the service you want to export
    consumers:
      peerName: cluster-01 ## The name of the peer that receives the service

exportedsvc.yml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ExportedServices
metadata:
  name: default ## The name of the partition containing the service
spec:
  services:
    name: backend-service ## The name of the service you want to export
    consumers:
      peerName: cluster-01 ## The name of the peer that receives the service

Create service intentions for the second cluster.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: backend-deny
spec:
  destination:
    name: backend-service
  sources:
   - name: "*"
     action: deny
   - name: frontend-service
     action: allow

intention.yml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: backend-deny
spec:
  destination:
    name: backend-service
  sources:
   - name: "*"
     action: deny
   - name: frontend-service
     action: allow

Apply the service file, the ExportedServices resource, and the intentions to the second cluster.

$ kubectl apply --filename backend-service.yml --filename exportedsvc.yml --filename intention.yml

$ kubectl apply --filename backend-service.yml --filename exportedsvc.yml --filename intention.yml

To confirm that you peered your clusters, in “cluster-01,” query the /health HTTP endpoint.

$ curl "localhost:8500/v1/health/connect/backend?peer=cluster-02"

$ curl "localhost:8500/v1/health/connect/backend?peer=cluster-02"

For the services in “cluster-01” that you want to access the “backend-service,” add the following annotations to the service file.

##…
annotations:
  "consul.hashicorp.com/connect-inject": "true"
  "consul.hashicorp.com/transparent-proxy": "false"
  "consul.hashicorp.com/connect-service-upstreams": "backend-service.svc.cluster-02.peer:1234"
##…

frontend-service.yml

##…
annotations:
  "consul.hashicorp.com/connect-inject": "true"
  "consul.hashicorp.com/transparent-proxy": "false"
  "consul.hashicorp.com/connect-service-upstreams": "backend-service.svc.cluster-02.peer:1234"
##…

Apply the service file to the first cluster.

$ kubectl apply --filename frontend-service.yml

$ kubectl apply --filename frontend-service.yml

Run the following command and check the output to confirm that you peered your clusters successfully.

$ curl localhost:1234
{
    “name”: “backend-service”,
    ##…
    “body”: “Response from backend”,
    “code”: 200
}

$ curl localhost:1234
{
    “name”: “backend-service”,
    ##…
    “body”: “Response from backend”,
    “code”: 200
}

End a peering connection

To end a peering connection, delete both the PeeringAcceptor and PeeringDialer resources.

To confirm that you deleted your peering connection, in “cluster-01,” query the /health HTTP endpoint. The peered services should no longer appear.

$ curl "localhost:8500/v1/health/connect/backend?peer=cluster-02"

$ curl "localhost:8500/v1/health/connect/backend?peer=cluster-02"