Service Affinity

This tutorial will guide you to enable service affinity across multiple Kubernetes clusters.

Prerequisites

You need to have a functioning Cluster Mesh with a Global Service, please follow the guide Setting up Cluster Mesh and Load-balancing & Service Discovery to set it up.

Enabling Global Service Affinity

Load-balancing across multiple clusters might not be ideal in some cases. The annotation io.cilium/service-affinity: "local|remote|none" can be used to specify the preferred endpoint destination.

For example, if the value of annotation io.cilium/service-affinity is local, the Global Service will load-balance across healthy local backends, and only user remote endpoints if and only if all of local backends are not available or unhealthy.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: rebel-base
  5.   annotations:
  6.      io.cilium/global-service: "true"
  7.      # Possible values:
  8.      # - local
  9.      #    preferred endpoints from local cluster if available
  10.      # - remote
  11.      #    preferred endpoints from remote cluster if available
  12.      # none (default)
  13.      #    no preference. Default behavior if this annotation does not exist
  14.      io.cilium/service-affinity: "local"
  15. spec:
  16.   type: ClusterIP
  17.   ports:
  18.   - port: 80
  19.   selector:
  20.     name: rebel-base

  1. In cluster 1, add io.cilium/service-affinity="local" to existing global service

    1. kubectl annotate service rebel-base io.cilium/service-affinity=local --overwrite

  2. From cluster 1, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in cluster 1 only.

  3. From cluster 2, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in both clusters as usual.

  4. From cluster 1, check the service endpoints, the local endpoints are marked as preferred.

    1. kubectl exec -n kube-system -ti ds/cilium -- cilium service list --clustermesh-affinity
    3. ID   Frontend            Service Type   Backend
    4. 1    10.96.0.1:443       ClusterIP      1 => 172.18.0.3:6443 (active)
    5. 2    10.96.0.10:53       ClusterIP      1 => 10.244.1.171:53 (active)
    6.                                         2 => 10.244.2.206:53 (active)
    7. 3    10.96.0.10:9153     ClusterIP      1 => 10.244.1.171:9153 (active)
    8.                                         2 => 10.244.2.206:9153 (active)
    9. 4    10.96.210.49:2379   ClusterIP      1 => 10.244.2.216:2379 (active)
    10. 5    10.96.173.113:80    ClusterIP      1 => 10.244.2.136:80 (active)
    11.                                         2 => 10.244.1.61:80 (active) (preferred)
    12.                                         3 => 10.244.2.31:80 (active) (preferred)
    13.                                         4 => 10.244.2.200:80 (active)

  5. In cluster 1, change io.cilium/service-affinity value to remote for existing global service

    1. kubectl annotate service rebel-base io.cilium/service-affinity=remote --overwrite

  6. From cluster 1, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    This time, the replies are coming from pods in cluster 2 only.

  7. From cluster 1, check the service endpoints, now the remote endpoints are marked as preferred.

    1. kubectl exec -n kube-system -ti ds/cilium -- cilium service list --clustermesh-affinity
    3. ID   Frontend            Service Type   Backend
    4. 1    10.96.0.1:443       ClusterIP      1 => 172.18.0.3:6443 (active)
    5. 2    10.96.0.10:53       ClusterIP      1 => 10.244.1.171:53 (active)
    6.                                         2 => 10.244.2.206:53 (active)
    7. 3    10.96.0.10:9153     ClusterIP      1 => 10.244.1.171:9153 (active)
    8.                                         2 => 10.244.2.206:9153 (active)
    9. 4    10.96.210.49:2379   ClusterIP      1 => 10.244.2.216:2379 (active)
    10. 5    10.96.173.113:80    ClusterIP      1 => 10.244.2.136:80 (active) (preferred)
    11.                                         2 => 10.244.1.61:80 (active)
    12.                                         3 => 10.244.2.31:80 (active)
    13.                                         4 => 10.244.2.200:80 (active) (preferred)

  8. From cluster 2, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in both clusters as usual.

  9. In cluster 1, remove io.cilium/service-affinity annotation for existing global service

    1. kubectl annotate service rebel-base io.cilium/service-affinity- --overwrite

  10. From either cluster, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in both clusters again.