BGP (beta)

BGP provides a way to advertise routes using traditional networking protocols to allow Cilium-managed services to be accessible outside the cluster.

This document explains how to configure Cilium’s native support for announcing LoadBalancer IPs of Services via BGP. It leverages MetalLB’s simple and effective implementation of IP allocation and the minimal BGP protocol support to do this. The configuration for Cilium is the same as MetalLB’s configuration.

Specifically, if a Service of type LoadBalancer is created, Cilium will allocate an IP for it from a specified pool. Once the IP is allocated, the Agents will announce via BGP depending on the Service’s ExternalTrafficPolicy. See MetalLB’s documentation on this specific topic.

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Deploy Cilium

Note

Make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

  1. helm repo add cilium https://helm.cilium.io/

BGP support is enabled by providing the BGP configuration via a ConfigMap and by setting a few Helm values. Otherwise, BGP is disabled by default.

Here’s an example ConfigMap:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: bgp-config
  5.   namespace: kube-system
  6. data:
  7.   config.yaml: |
  8.     peers:
  9.       - peer-address: 10.0.0.1
  10.         peer-asn: 64512
  11.         my-asn: 64512
  12.     address-pools:
  13.       - name: default
  14.         protocol: bgp
  15.         addresses:
  16.           - 192.0.2.0/24

Here are the required Helm values:

  1. helm install cilium cilium/cilium --version 1.10.2 --set bgp.enabled=true --set bgp.announce.loadbalancerIP=true

Verify that Cilium Agent pod is running.

  1. $ kubectl -n kube-system get pods -l k8s-app=cilium
  2. NAME           READY   STATUS    RESTARTS   AGE
  3. cilium-5ngzd   1/1     Running   0          3m19s

Create LoadBalancer and backend pods

Apply the following LoadBalancer Service and its corresponding backends:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: test-lb
  5. spec:
  6.   type: LoadBalancer
  7.   ports:
  8.   - port: 80
  9.     targetPort: 80
  10.     protocol: TCP
  11.     name: http
  12.   selector:
  13.     svc: test-lb
  14. ---
  15. apiVersion: apps/v1
  16. kind: Deployment
  17. metadata:
  18.   name: nginx
  19. spec:
  20.   selector:
  21.     matchLabels:
  22.       svc: test-lb
  23.   template:
  24.     metadata:
  25.       labels:
  26.         svc: test-lb
  27.     spec:
  28.       containers:
  29.       - name: web
  30.         image: nginx
  31.         imagePullPolicy: IfNotPresent
  32.         ports:
  33.         - containerPort: 80
  34.         readinessProbe:
  35.           httpGet:
  36.             path: /
  37.             port: 80

Observe that the Operator allocates an external IP for test-lb:

  1. $ kubectl get svc
  2. NAME        TYPE          CLUSTER-IP  EXTERNAL-IP  PORT(S)       AGE
  3. kubernetes  ClusterIP     172.20.0.1  <none>       443/TCP       4d23h
  4. test-lb     LoadBalancer  172.20.0.5  192.0.2.154  80:30724/TCP  10s

Verify that the backend is running:

  1. $ kubectl get pods | grep nginx
  2. nginx                      1/1     Running   0          16s

Validate BGP announcements

To see whether Cilium is announcing the external IP of the Service, check the node’s routing table that’s running your BGP router.

Alternatively, you can run tcpdump inside the Cilium pod (it’ll need to be apt install’d) to see BGP messages like so:

  1. root@kind-worker:/home/cilium# tcpdump -n -i any tcp port 179
  2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  3. listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
  4. 17:03:19.380682 IP 172.20.0.2.43261 > 10.0.0.1.179: Flags [P.], seq 2930402899:2930402918, ack 2731344744, win 502, options [nop,nop,TS val 4080796863 ecr 4108836857], length 19: BGP
  5. 17:03:19.385065 IP 10.0.0.1.179 > 172.20.0.2.43261: Flags [P.], seq 1:20, ack 19, win 509, options [nop,nop,TS val 4108866857 ecr 4080796863], length 19: BGP

Verify that traffic to the external IP is directed to the backends:

  1. $ # Exec / SSH into BGP router
  2. $ curl 192.0.2.154

Limitations

BGP support relies on MetalLB. Due to the lack of upstream support for EndpointSlices in MetalLB, Cilium will fallback to using the original Endpoints resource.

The Kubernetes documentation provides a simple explanation of the advantages of EndpointSlices and the issues with Endpoints.