Traefik & CRD & Let’s Encrypt

Traefik with an IngressRoute Custom Resource Definition for Kubernetes, and TLS Through Let’s Encrypt.

This document is intended to be a fully working example demonstrating how to set up Traefik in Kubernetes, with the dynamic configuration coming from the IngressRoute Custom Resource, and TLS setup with Let’s Encrypt. However, for the sake of simplicity, we’re using k3s docker image for the Kubernetes cluster setup.

Please note that for this setup, given that we’re going to use ACME’s TLS-ALPN-01 challenge, the host you’ll be running it on must be able to receive connections from the outside on port 443. And of course its internet facing IP address must match the domain name you intend to use.

In the following, the Kubernetes resources defined in YAML configuration files can be applied to the setup in two different ways:

  • the first, and usual way, is simply with the kubectl apply command.
  • the second, which can be used for this tutorial, is to directly place the files in the directory used by the k3s docker image for such inputs (/var/lib/rancher/k3s/server/manifests).

Kubectl Version

With the rancher/k3s version used in this guide (0.8.0), the kubectl version needs to be >= 1.11.

k3s Docker-compose Configuration

Our starting point is the docker-compose configuration file, to start the k3s cluster. You can start it with:

  1. docker-compose -f k3s.yml up
  1. server:
  2. image: rancher/k3s:v1.17.2-k3s1
  3. command: server --disable-agent --no-deploy traefik
  4. environment:
  5. - K3S_CLUSTER_SECRET=somethingtotallyrandom
  6. - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
  7. - K3S_KUBECONFIG_MODE=666
  8. volumes:
  9. # k3s will generate a kubeconfig.yaml in this directory. This volume is mounted
  10. # on your host, so you can then 'export KUBECONFIG=/somewhere/on/your/host/out/kubeconfig.yaml',
  11. # in order for your kubectl commands to work.
  12. - /somewhere/on/your/host/out:/output
  13. # This directory is where you put all the (yaml) configuration files of
  14. # the Kubernetes resources.
  15. - /somewhere/on/your/host/in:/var/lib/rancher/k3s/server/manifests
  16. ports:
  17. - 6443:6443
  18. node:
  19. image: rancher/k3s:v1.17.2-k3s1
  20. privileged: true
  21. links:
  22. - server
  23. environment:
  24. - K3S_URL=https://server:6443
  25. - K3S_CLUSTER_SECRET=somethingtotallyrandom
  26. volumes:
  27. # this is where you would place a alternative traefik image (saved as a .tar file with
  28. # 'docker save'), if you want to use it, instead of the traefik:v2.4 image.
  29. - /sowewhere/on/your/host/custom-image:/var/lib/rancher/k3s/agent/images

Cluster Resources

Let’s now have a look (in the order they should be applied, if using kubectl apply) at all the required resources for the full setup.

IngressRoute Definition

First, the definition of the IngressRoute and the Middleware kinds. Also note the RBAC authorization resources; they’ll be referenced through the serviceAccountName of the deployment, later on.

  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4. name: ingressroutes.traefik.containo.us
  5. spec:
  6. group: traefik.containo.us
  7. version: v1alpha1
  8. names:
  9. kind: IngressRoute
  10. plural: ingressroutes
  11. singular: ingressroute
  12. scope: Namespaced
  13. ---
  14. apiVersion: apiextensions.k8s.io/v1beta1
  15. kind: CustomResourceDefinition
  16. metadata:
  17. name: middlewares.traefik.containo.us
  18. spec:
  19. group: traefik.containo.us
  20. version: v1alpha1
  21. names:
  22. kind: Middleware
  23. plural: middlewares
  24. singular: middleware
  25. scope: Namespaced
  26. ---
  27. apiVersion: apiextensions.k8s.io/v1beta1
  28. kind: CustomResourceDefinition
  29. metadata:
  30. name: ingressroutetcps.traefik.containo.us
  31. spec:
  32. group: traefik.containo.us
  33. version: v1alpha1
  34. names:
  35. kind: IngressRouteTCP
  36. plural: ingressroutetcps
  37. singular: ingressroutetcp
  38. scope: Namespaced
  39. ---
  40. apiVersion: apiextensions.k8s.io/v1beta1
  41. kind: CustomResourceDefinition
  42. metadata:
  43. name: ingressrouteudps.traefik.containo.us
  44. spec:
  45. group: traefik.containo.us
  46. version: v1alpha1
  47. names:
  48. kind: IngressRouteUDP
  49. plural: ingressrouteudps
  50. singular: ingressrouteudp
  51. scope: Namespaced
  52. ---
  53. apiVersion: apiextensions.k8s.io/v1beta1
  54. kind: CustomResourceDefinition
  55. metadata:
  56. name: tlsoptions.traefik.containo.us
  57. spec:
  58. group: traefik.containo.us
  59. version: v1alpha1
  60. names:
  61. kind: TLSOption
  62. plural: tlsoptions
  63. singular: tlsoption
  64. scope: Namespaced
  65. ---
  66. apiVersion: apiextensions.k8s.io/v1beta1
  67. kind: CustomResourceDefinition
  68. metadata:
  69. name: tlsstores.traefik.containo.us
  70. spec:
  71. group: traefik.containo.us
  72. version: v1alpha1
  73. names:
  74. kind: TLSStore
  75. plural: tlsstores
  76. singular: tlsstore
  77. scope: Namespaced
  78. ---
  79. apiVersion: apiextensions.k8s.io/v1beta1
  80. kind: CustomResourceDefinition
  81. metadata:
  82. name: traefikservices.traefik.containo.us
  83. spec:
  84. group: traefik.containo.us
  85. version: v1alpha1
  86. names:
  87. kind: TraefikService
  88. plural: traefikservices
  89. singular: traefikservice
  90. scope: Namespaced
  91. ---
  92. apiVersion: apiextensions.k8s.io/v1beta1
  93. kind: CustomResourceDefinition
  94. metadata:
  95. name: serverstransports.traefik.containo.us
  96. spec:
  97. group: traefik.containo.us
  98. version: v1alpha1
  99. names:
  100. kind: ServersTransport
  101. plural: serverstransports
  102. singular: serverstransport
  103. scope: Namespaced
  104. ---
  105. kind: ClusterRole
  106. apiVersion: rbac.authorization.k8s.io/v1beta1
  107. metadata:
  108. name: traefik-ingress-controller
  109. rules:
  110. - apiGroups:
  111. - ""
  112. resources:
  113. - services
  114. - endpoints
  115. - secrets
  116. verbs:
  117. - get
  118. - list
  119. - watch
  120. - apiGroups:
  121. - extensions
  122. - networking.k8s.io
  123. resources:
  124. - ingresses
  125. - ingressclasses
  126. verbs:
  127. - get
  128. - list
  129. - watch
  130. - apiGroups:
  131. - extensions
  132. resources:
  133. - ingresses/status
  134. verbs:
  135. - update
  136. - apiGroups:
  137. - traefik.containo.us
  138. resources:
  139. - middlewares
  140. - ingressroutes
  141. - traefikservices
  142. - ingressroutetcps
  143. - ingressrouteudps
  144. - tlsoptions
  145. - tlsstores
  146. - serverstransports
  147. verbs:
  148. - get
  149. - list
  150. - watch
  151. ---
  152. kind: ClusterRoleBinding
  153. apiVersion: rbac.authorization.k8s.io/v1beta1
  154. metadata:
  155. name: traefik-ingress-controller
  156. roleRef:
  157. apiGroup: rbac.authorization.k8s.io
  158. kind: ClusterRole
  159. name: traefik-ingress-controller
  160. subjects:
  161. - kind: ServiceAccount
  162. name: traefik-ingress-controller
  163. namespace: default

Services

Then, the services. One for Traefik itself, and one for the app it routes for, i.e. in this case our demo HTTP server: whoami.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. name: traefik
  5. spec:
  6. ports:
  7. - protocol: TCP
  8. name: web
  9. port: 8000
  10. - protocol: TCP
  11. name: admin
  12. port: 8080
  13. - protocol: TCP
  14. name: websecure
  15. port: 4443
  16. selector:
  17. app: traefik
  18. ---
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22. name: whoami
  23. spec:
  24. ports:
  25. - protocol: TCP
  26. name: web
  27. port: 80
  28. selector:
  29. app: whoami

Deployments

Next, the deployments, i.e. the actual pods behind the services. Again, one pod for Traefik, and one for the whoami app.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4. namespace: default
  5. name: traefik-ingress-controller
  6. ---
  7. kind: Deployment
  8. apiVersion: apps/v1
  9. metadata:
  10. namespace: default
  11. name: traefik
  12. labels:
  13. app: traefik
  14. spec:
  15. replicas: 1
  16. selector:
  17. matchLabels:
  18. app: traefik
  19. template:
  20. metadata:
  21. labels:
  22. app: traefik
  23. spec:
  24. serviceAccountName: traefik-ingress-controller
  25. containers:
  26. - name: traefik
  27. image: traefik:v2.4
  28. args:
  29. - --api.insecure
  30. - --accesslog
  31. - --entrypoints.web.Address=:8000
  32. - --entrypoints.websecure.Address=:4443
  33. - --providers.kubernetescrd
  34. - --certificatesresolvers.myresolver.acme.tlschallenge
  35. - --certificatesresolvers.myresolver.acme.email=foo@you.com
  36. - --certificatesresolvers.myresolver.acme.storage=acme.json
  37. # Please note that this is the staging Let's Encrypt server.
  38. # Once you get things working, you should remove that whole line altogether.
  39. - --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
  40. ports:
  41. - name: web
  42. containerPort: 8000
  43. - name: websecure
  44. containerPort: 4443
  45. - name: admin
  46. containerPort: 8080
  47. ---
  48. kind: Deployment
  49. apiVersion: apps/v1
  50. metadata:
  51. namespace: default
  52. name: whoami
  53. labels:
  54. app: whoami
  55. spec:
  56. replicas: 2
  57. selector:
  58. matchLabels:
  59. app: whoami
  60. template:
  61. metadata:
  62. labels:
  63. app: whoami
  64. spec:
  65. containers:
  66. - name: whoami
  67. image: traefik/whoami
  68. ports:
  69. - name: web
  70. containerPort: 80

Port Forwarding

Now, as an exception to what we said above, please note that you should not let the ingressRoute resources below be applied automatically to your cluster. The reason is, as soon as the ACME provider of Traefik detects we have TLS routers, it will try to generate the certificates for the corresponding domains. And this will not work, because as it is, our Traefik pod is not reachable from the outside, which will make the ACME TLS challenge fail. Therefore, for the whole thing to work, we must delay applying the ingressRoute resources until we have port-forwarding set up properly, which is the next step.

  1. kubectl port-forward --address 0.0.0.0 service/traefik 8000:8000 8080:8080 443:4443 -n default

Also, and this is out of the scope if this guide, please note that because of the privileged ports limitation on Linux, the above command might fail to listen on port 443. In which case you can use tricks such as elevating caps of kubectl with setcaps, or using authbind, or setting up a NAT between your host and the WAN. Look it up.

Traefik Routers

We can now finally apply the actual ingressRoutes, with:

  1. kubectl apply -f 04-ingressroutes.yml
  1. apiVersion: traefik.containo.us/v1alpha1
  2. kind: IngressRoute
  3. metadata:
  4. name: simpleingressroute
  5. namespace: default
  6. spec:
  7. entryPoints:
  8. - web
  9. routes:
  10. - match: Host(`your.example.com`) && PathPrefix(`/notls`)
  11. kind: Rule
  12. services:
  13. - name: whoami
  14. port: 80
  15. ---
  16. apiVersion: traefik.containo.us/v1alpha1
  17. kind: IngressRoute
  18. metadata:
  19. name: ingressroutetls
  20. namespace: default
  21. spec:
  22. entryPoints:
  23. - websecure
  24. routes:
  25. - match: Host(`your.example.com`) && PathPrefix(`/tls`)
  26. kind: Rule
  27. services:
  28. - name: whoami
  29. port: 80
  30. tls:
  31. certResolver: myresolver

Give it a few seconds for the ACME TLS challenge to complete, and you should then be able to access your whoami pod (routed through Traefik), from the outside. Both with or (just for fun, do not do that in production) without TLS:

  1. curl [-k] https://your.example.com/tls
  1. curl http://your.example.com:8000/notls

Note that you’ll have to use -k as long as you’re using the staging server of Let’s Encrypt, since it is not an authorized certificate authority on systems where it hasn’t been manually added.