Using ExternalDNS on Google Cloud Platform to automate DNS setup

ExternalDNS is a tool that synchronizes exposed Kubernetes Services and Ingresses with DNS providers.

This doc explains how to set up ExternalDNS within a Knative cluster using Google Cloud DNS to automate the process of publishing the Knative domain.

Set up environtment variables

Run the following command to configure the environment variables

  1. export PROJECT_NAME=<your-google-cloud-project-name>
  3. export CUSTOM_DOMAIN=<your-custom-domain-used-in-knative>
  5. export CLUSTER_NAME=<knative-cluster-name>
  7. export CLUSTER_ZONE=<knative-cluster-zone>

Set up Kubernetes Engine cluster with CloudDNS read/write permissions

There are two ways to set up a Kubernetes Engine cluster with CloudDNS read/write permissions.

Cluster with Cloud DNS scope

You can create a GKE cluster with Cloud DNS scope by entering the following command:

  1. gcloud container clusters create $CLUSTER_NAME \
  2.     --zone=$CLUSTER_ZONE \
  3.     --cluster-version=latest \
  4.     --machine-type=n1-standard-4 \
  5.     --enable-autoscaling --min-nodes=1 --max-nodes=10 \
  6.     --enable-autorepair \
  7.     --scopes=service-control,service-management,compute-rw,storage-ro,cloud-platform,logging-write,monitoring-write,pubsub,datastore,"https://www.googleapis.com/auth/ndev.clouddns.readwrite" \
  8.     --num-nodes=3

Note that by using this way, any pod within the cluster will have permissions to read/write CloudDNS.

Cluster with Cloud DNS Admin Service Account credential

  1. Create a GKE cluster without Cloud DNS scope by entering the following command:
  1. gcloud container clusters create $CLUSTER_NAME \
  2.     --zone=$CLUSTER_ZONE \
  3.     --cluster-version=latest \
  4.     --machine-type=n1-standard-4 \
  5.     --enable-autoscaling --min-nodes=1 --max-nodes=10 \
  6.     --enable-autorepair \
  7.     --scopes=service-control,service-management,compute-rw,storage-ro,cloud-platform,logging-write,monitoring-write,pubsub,datastore \
  8.     --num-nodes=3
  1. Create a new service account for Cloud DNS admin role.
  1. # Name of the service account you want to create.
  2. export CLOUD_DNS_SA=cloud-dns-admin
  4. gcloud --project $PROJECT_NAME iam service-accounts \
  5.     create $CLOUD_DNS_SA \
  6.     --display-name "Service Account to support ACME DNS-01 challenge."
  1. Bind the role dns.admin to the newly created service account.
  1. # Fully-qualified service account name also has project-id information.
  2. export CLOUD_DNS_SA=$CLOUD_DNS_SA@$PROJECT_NAME.iam.gserviceaccount.com
  4. gcloud projects add-iam-policy-binding $PROJECT_NAME \
  5.     --member serviceAccount:$CLOUD_DNS_SA \
  6.     --role roles/dns.admin
  1. Download the secret key file for your service account.
  1. gcloud iam service-accounts keys create ~/key.json \
  2.     --iam-account=$CLOUD_DNS_SA
  1. Upload the service account credential to your cluster. This command uses the secret name cloud-dns-key, but you can choose a different name.
  1. kubectl create secret generic cloud-dns-key \
  2.     --from-file=key.json=$HOME/key.json
  1. Delete the local secret
  1. rm ~/key.json

Now your cluster has the credential of your CloudDNS admin service account. And it can be used to access your Cloud DNS. You can enforce the access of the credentail secret within your cluster, so that only the pods that have the permission to get the credential secret can access your Cloud DNS.

Set up Knative

  1. Follow the instruction to install Knative on your cluster.
  2. Configure Knative to use your custom domain.
  1. kubectl edit cm config-domain --namespace knative-serving

This command opens your default text editor and allows you to edit the config map.

  1. apiVersion: v1
  2. data:
  3.   example.com: ""
  4. kind: ConfigMap
  5. [...]

Edit the file to replace example.com with your custom domain (the value of $CUSTOM_DOMAIN) and save your changes. In this example, we use domain external-dns-test.my-org.do for all routes:

  1. apiVersion: v1
  2. data:
  3.   external-dns-test.my-org.do: ""
  4. kind: ConfigMap
  5. [...]

Set up ExternalDNS

This guide uses Google Cloud Platform as an example to show how to set up ExternalDNS. You can find detailed instructions for other cloud providers in the ExternalDNS documentation.

Create a DNS zone for managing DNS records

Skip this step if you already have a zone for managing the DNS records of your custom domain.

A DNS zone which will contain the managed DNS records needs to be created.

Use the following command to create a DNS zone with Google Cloud DNS:

  1. export DNS_ZONE_NAME=<dns-zone-name>
  3. gcloud dns managed-zones create $DNS_ZONE_NAME \
  4.     --dns-name $CUSTOM_DOMAIN \
  5.     --description "Automatically managed zone by kubernetes.io/external-dns"

Make a note of the nameservers that were assigned to your new zone.

  1. gcloud dns record-sets list \
  2.     --zone $DNS_ZONE_NAME \
  3.     --name $CUSTOM_DOMAIN \
  4.     --type NS

You should see output similar to the following assuming your custom domain is external-dns-test.my-org.do:

  1. NAME                             TYPE  TTL    DATA
  2. external-dns-test.my-org.do.  NS    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.

In this case, the DNS nameservers are ns-cloud-{e1-e4}.googledomains.com. Yours could differ slightly, e.g. {a1-a4}, {b1-b4} etc.

If this zone has the parent zone, you need to add NS records of this zone into the parent zone so that this zone can be found from the parent. Assuming the parent zone is my-org-do and the parent domain is my-org.do, and the parent zone is also hosted at Google Cloud DNS, you can follow these steps to add the NS records of this zone into the parent zone:

  1. gcloud dns record-sets transaction start --zone "my-org-do"
  2. gcloud dns record-sets transaction add ns-cloud-e{1..4}.googledomains.com. \
  3.     --name "external-dns-test.my-org.do." --ttl 300 --type NS --zone "my-org-do"
  4. gcloud dns record-sets transaction execute --zone "my-org-do"

Deploy ExternalDNS

Firstly, choose the manifest of ExternalDNS.

Use below manifest if you set up your cluster with CloudDNS scope.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: external-dns
  5. ---
  6. apiVersion: rbac.authorization.k8s.io/v1
  7. kind: ClusterRole
  8. metadata:
  9.   name: external-dns
  10. rules:
  11.   - apiGroups: [""]
  12.     resources: ["services"]
  13.     verbs: ["get", "watch", "list"]
  14.   - apiGroups: [""]
  15.     resources: ["pods"]
  16.     verbs: ["get", "watch", "list"]
  17.   - apiGroups: ["extensions"]
  18.     resources: ["ingresses"]
  19.     verbs: ["get", "watch", "list"]
  20.   - apiGroups: [""]
  21.     resources: ["nodes"]
  22.     verbs: ["list"]
  23. ---
  24. apiVersion: rbac.authorization.k8s.io/v1
  25. kind: ClusterRoleBinding
  26. metadata:
  27.   name: external-dns-viewer
  28. roleRef:
  29.   apiGroup: rbac.authorization.k8s.io
  30.   kind: ClusterRole
  31.   name: external-dns
  32. subjects:
  33.   - kind: ServiceAccount
  34.     name: external-dns
  35.     namespace: default
  36. ---
  37. apiVersion: extensions/v1beta1
  38. kind: Deployment
  39. metadata:
  40.   name: external-dns
  41. spec:
  42.   strategy:
  43.     type: Recreate
  44.   template:
  45.     metadata:
  46.       labels:
  47.         app: external-dns
  48.     spec:
  49.       serviceAccountName: external-dns
  50.       containers:
  51.         - name: external-dns
  52.           image: registry.opensource.zalan.do/teapot/external-dns:latest
  53.           args:
  54.             - --source=service
  55.             - --domain-filter=$CUSTOM_DOMAIN # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
  56.             - --provider=google
  57.             - --google-project=$PROJECT_NAME # Use this to specify a project different from the one external-dns is running inside
  58.             - --policy=sync # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
  59.             - --registry=txt
  60.             - --txt-owner-id=my-identifier

Or use below manifest if you set up your cluster with CloudDNS service account credential.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: external-dns
  5. ---
  6. apiVersion: rbac.authorization.k8s.io/v1
  7. kind: ClusterRole
  8. metadata:
  9.   name: external-dns
  10. rules:
  11.   - apiGroups: [""]
  12.     resources: ["services"]
  13.     verbs: ["get", "watch", "list"]
  14.   - apiGroups: [""]
  15.     resources: ["pods,secrets"]
  16.     verbs: ["get", "watch", "list"]
  17.   - apiGroups: ["extensions"]
  18.     resources: ["ingresses"]
  19.     verbs: ["get", "watch", "list"]
  20.   - apiGroups: [""]
  21.     resources: ["nodes"]
  22.     verbs: ["list"]
  23. ---
  24. apiVersion: rbac.authorization.k8s.io/v1
  25. kind: ClusterRoleBinding
  26. metadata:
  27.   name: external-dns-viewer
  28. roleRef:
  29.   apiGroup: rbac.authorization.k8s.io
  30.   kind: ClusterRole
  31.   name: external-dns
  32. subjects:
  33.   - kind: ServiceAccount
  34.     name: external-dns
  35.     namespace: default
  36. ---
  37. apiVersion: extensions/v1beta1
  38. kind: Deployment
  39. metadata:
  40.   name: external-dns
  41. spec:
  42.   strategy:
  43.     type: Recreate
  44.   template:
  45.     metadata:
  46.       labels:
  47.         app: external-dns
  48.     spec:
  49.       volumes:
  50.         - name: google-cloud-key
  51.           secret:
  52.             secretName: cloud-dns-key
  53.       serviceAccountName: external-dns
  54.       containers:
  55.         - name: external-dns
  56.           image: registry.opensource.zalan.do/teapot/external-dns:latest
  57.           volumeMounts:
  58.             - name: google-cloud-key
  59.               mountPath: /var/secrets/google
  60.           env:
  61.             - name: GOOGLE_APPLICATION_CREDENTIALS
  62.               value: /var/secrets/google/key.json
  63.           args:
  64.             - --source=service
  65.             - --domain-filter=$CUSTOM_DOMAIN # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
  66.             - --provider=google
  67.             - --google-project=$PROJECT_NAME # Use this to specify a project different from the one external-dns is running inside
  68.             - --policy=sync # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
  69.             - --registry=txt
  70.             - --txt-owner-id=my-identifier

Then use the following command to apply the manifest you chose to install ExternalDNS

  1. cat <<EOF | kubectl apply --filename -
  2. <your-chosen-manifest>
  3. EOF

You should see ExternalDNS is installed by running:

  1. kubectl get deployment external-dns

Configuring Knative Gateway service

In order to publish the Knative Gateway service, the annotation external-dns.alpha.kubernetes.io/hostname: '*.$CUSTOM_DOMAIN needs to be added into Knative gateway service:

  1. INGRESSGATEWAY=istio-ingressgateway
  3. kubectl edit svc $INGRESSGATEWAY --namespace istio-system

This command opens your default text editor and allows you to add the annotation to istio-ingressgateway service. After you’ve added your annotation, your file may look similar to this (assuming your custom domain is external-dns-test.my-org.do):

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   annotations:
  5.     external-dns.alpha.kubernetes.io/hostname: '*.external-dns-test.my-org.do'
  6.     ...

Verify ExternalDNS works

After roughly two minutes, check that a corresponding DNS record for your service was created.

  1. gcloud dns record-sets list     --zone $DNS_ZONE_NAME     --name "*.$CUSTOM_DOMAIN."

You should see output similar to:

  1. NAME                            TYPE  TTL  DATA
  2. *.external-dns-test.my-org.do.  A     300  35.231.248.30
  3. *.external-dns-test.my-org.do.  TXT   300  "heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=service/istio-system/istio-ingressgateway"

Verify domain has been published

You can check if the domain has been published to the Internet be entering the following command:

  1. host test.external-dns-test.my-org.do

You should see the below result after the domain is published:

  1. test.external-dns-test.my-org.do has address 35.231.248.30

Note: The process of publishing the domain to the Internet can take several minutes.