Locking down external access using AWS metadata

This document serves as an introduction to using Cilium to enforce policies based on AWS instances metadata. It is a detailed walk-through of getting a single-node Cilium environment running on your machine. It is designed to take 15-30 minutes with some experience running Kubernetes.

Setup Cilium

This guide will work with any approach to installing Cilium, including minikube, as long as the cilium-operator pod in the deployment can reach the AWS API server However, since the most common use of this mechanism is for Kubernetes clusters running in AWS, we recommend trying it out along with the guide: Quick Installation .

Create AWS secrets

Before installing Cilium, a new Kubernetes Secret with the AWS Tokens needs to be added to your Kubernetes cluster. This Secret will allow Cilium to gather information from the AWS API which is needed to implement ToGroups policies.

AWS Access keys and IAM role

To create a new access token the following guide can be used. These keys need to have certain permissions set:

  1. {
  2. "Version": "2012-10-17",
  3. "Statement": [
  4. {
  5. "Effect": "Allow",
  6. "Action": "ec2:Describe*",
  7. "Resource": "*"
  8. }
  9. ]
  10. }

As soon as you have the access tokens, the following secret needs to be added, with each empty string replaced by the associated value as a base64-encoded string:

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4. name: cilium-aws
  5. namespace: kube-system
  6. type: Opaque
  7. data:
  8. AWS_ACCESS_KEY_ID: ""
  9. AWS_SECRET_ACCESS_KEY: ""
  10. AWS_DEFAULT_REGION: ""

The base64 command line utility can be used to generate each value, for example:

  1. $ echo -n "eu-west-1" | base64
  2. ZXUtd2VzdC0x

This secret stores the AWS credentials, which will be used to connect the AWS API.

  1. $ kubectl create -f cilium-secret.yaml

To validate that the credentials are correct, the following pod can be created for debugging purposes:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. name: testing-aws-pod
  5. namespace: kube-system
  6. spec:
  7. containers:
  8. - name: aws-cli
  9. image: mesosphere/aws-cli
  10. command: ['sh', '-c', 'sleep 3600']
  11. env:
  12. - name: AWS_ACCESS_KEY_ID
  13. valueFrom:
  14. secretKeyRef:
  15. name: cilium-aws
  16. key: AWS_ACCESS_KEY_ID
  17. optional: true
  18. - name: AWS_SECRET_ACCESS_KEY
  19. valueFrom:
  20. secretKeyRef:
  21. name: cilium-aws
  22. key: AWS_SECRET_ACCESS_KEY
  23. optional: true
  24. - name: AWS_DEFAULT_REGION
  25. valueFrom:
  26. secretKeyRef:
  27. name: cilium-aws
  28. key: AWS_DEFAULT_REGION
  29. optional: true

To list all of the available AWS instances, the following command can be used:

  1. $ kubectl -n kube-system exec -ti testing-aws-pod -- aws ec2 describe-instances

Once the secret has been created and validated, the cilium-operator pod must be restarted in order to pick up the credentials in the secret. To do this, identify and delete the existing cilium-operator pod, which will be recreated automatically:

  1. $ kubectl get pods -l name=cilium-operator -n kube-system
  2. NAME READY STATUS RESTARTS AGE
  3. cilium-operator-7c9d69f7c-97vqx 1/1 Running 0 36h
  4. $ kubectl delete pod cilium-operator-7c9d69f7c-97vqx

It is important for this demo that coredns is working correctly. To know the status of coredns you can run the following command:

  1. $ kubectl get deployment kube-dns -n kube-system
  2. NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
  3. coredns 2 2 2 2 13h

Where at least one pod should be available.

Configure AWS Security Groups

Cilium’s AWS Metadata filtering capability enables explicit whitelisting of communication between a subset of pods (identified by Kubernetes labels) with a set of destination EC2 VMs (identified by membership in an AWS security group).

In this example, the destination EC2 VMs are a member of a single AWS security group (‘sg-0f2146100a88d03c3’) and pods with label class=xwing should only be able to make connections outside the cluster to the destination VMs in that security group.

To enable this, the VMs acting as Kubernetes worker nodes must be able to send traffic to the destination VMs that are being accessed by pods. One approach for achieving this is to put all Kubernetes worker VMs in a single ‘k8s-worker’ security group, and then ensure that any security group that is referenced in a Cilium toGroups policy has an allow all ingress rule (all ports) for connections from the ‘k8s-worker’ security group. Cilium filtering will then ensure that the only pods allowed by policy can reach the destination VMs.

Create a sample policy

Deploy a demo application:

In this case we’re going to use a demo application that is used in other guides. These manifests will create three microservices applications: deathstar, tiefighter, and xwing. In this case, we are only going to use our xwing microservice to secure communications to existing AWS instances.

  1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.10/examples/minikube/http-sw-app.yaml
  2. service "deathstar" created
  3. deployment "deathstar" created
  4. deployment "tiefighter" created
  5. deployment "xwing" created

Kubernetes will deploy the pods and service in the background. Running kubectl get pods,svc will inform you about the progress of the operation. Each pod will go through several states until it reaches Running at which point the pod is ready.

  1. $ kubectl get pods,svc
  2. NAME READY STATUS RESTARTS AGE
  3. po/deathstar-76995f4687-2mxb2 1/1 Running 0 1m
  4. po/deathstar-76995f4687-xbgnl 1/1 Running 0 1m
  5. po/tiefighter 1/1 Running 0 1m
  6. po/xwing 1/1 Running 0 1m
  7. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  8. svc/deathstar ClusterIP 10.109.254.198 <none> 80/TCP 3h
  9. svc/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h

Policy Language:

ToGroups rules can be used to define policy in relation to cloud providers, like AWS.

  1. ---
  2. kind: CiliumNetworkPolicy
  3. apiVersion: cilium.io/v2
  4. metadata:
  5. name: to-groups-sample
  6. namespace: default
  7. spec:
  8. endpointSelector:
  9. matchLabels:
  10. org: alliance
  11. class: xwing
  12. egress:
  13. - toPorts:
  14. - ports:
  15. - port: '80'
  16. protocol: TCP
  17. toGroups:
  18. - aws:
  19. securityGroupsIds:
  20. - 'sg-0f2146100a88d03c3'

This policy allows traffic from pod xwing to any AWS instance that is in the security group with ID sg-0f2146100a88d03c3.

Validate that derived policy is in place

Every time that a new policy with ToGroups rules is added, an equivalent policy (also called “derivative policy”), will be created. This policy will contain the set of CIDRs that correspond to the specification in ToGroups, e.g., the IPs of all instances that are part of a specified security group. The list of IPs will be updated periodically.

  1. $ kubectl get cnp
  2. NAME AGE
  3. to-groups-sample 11s
  4. to-groups-sample-togroups-044ba7d1-f491-11e8-ad2e-080027d2d952 10s

Eventually, the derivative policy will contain IPs in the ToCIDR section:

  1. $ kubectl get cnp to-groups-sample-togroups-044ba7d1-f491-11e8-ad2e-080027d2d952
  1. apiVersion: cilium.io/v2
  2. kind: CiliumNetworkPolicy
  3. metadata:
  4. creationTimestamp: 2018-11-30T11:13:52Z
  5. generation: 1
  6. labels:
  7. io.cilium.network.policy.kind: derivative
  8. io.cilium.network.policy.parent.uuid: 044ba7d1-f491-11e8-ad2e-080027d2d952
  9. name: to-groups-sample-togroups-044ba7d1-f491-11e8-ad2e-080027d2d952
  10. namespace: default
  11. ownerReferences:
  12. - apiVersion: cilium.io/v2
  13. blockOwnerDeletion: true
  14. kind: CiliumNetworkPolicy
  15. name: to-groups-sample
  16. uid: 044ba7d1-f491-11e8-ad2e-080027d2d952
  17. resourceVersion: "34853"
  18. selfLink: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/to-groups-sample-togroups-044ba7d1-f491-11e8-ad2e-080027d2d952
  19. uid: 04b289ba-f491-11e8-ad2e-080027d2d952
  20. specs:
  21. - egress:
  22. - toCIDRSet:
  23. - cidr: 34.254.113.42/32
  24. - cidr: 172.31.44.160/32
  25. toPorts:
  26. - ports:
  27. - port: "80"
  28. protocol: TCP
  29. endpointSelector:
  30. matchLabels:
  31. any:class: xwing
  32. any:org: alliance
  33. k8s:io.kubernetes.pod.namespace: default
  34. labels:
  35. - key: io.cilium.k8s.policy.name
  36. source: k8s
  37. value: to-groups-sample
  38. - key: io.cilium.k8s.policy.uid
  39. source: k8s
  40. value: 044ba7d1-f491-11e8-ad2e-080027d2d952
  41. - key: io.cilium.k8s.policy.namespace
  42. source: k8s
  43. value: default
  44. - key: io.cilium.k8s.policy.derived-from
  45. source: k8s
  46. value: CiliumNetworkPolicy
  47. status:
  48. nodes:
  49. k8s1:
  50. enforcing: true
  51. lastUpdated: 2018-11-30T11:28:03.907678888Z
  52. localPolicyRevision: 28
  53. ok: true

The derivative rule should contain the following information:

  • metadata.OwnerReferences: that contains the information about the ToGroups policy.
  • specs.Egress.ToCIDRSet: the list of private and public IPs of the instances that correspond to the spec of the parent policy.
  • status: whether or not the policy is enforced yet, and when the policy was last updated.

The Cilium Endpoint status for the xwing should have policy enforcement enabled only for egress connectivity:

  1. $ kubectl get cep xwing
  2. NAME ENDPOINT ID IDENTITY ID POLICY ENFORCEMENT ENDPOINT STATE IPV4 IPV6
  3. xwing 23453 63929 egress ready 10.10.0.95 f00d::a0a:0:0:22cf

In this example, xwing pod can only connect to 34.254.113.42/32 and 172.31.44.160/32 and connectivity to other IP will be denied.