Kubernetes Admission Control

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations. With OPA you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

Goals

This tutorial shows how to enforce custom policies on Kubernetes objects using OPA. In this tutorial, you will define admission control rules that prevent users from creating Kubernetes Ingress objects that violate the following organization policy:

  • Ingress hostnames must be whitelisted on the Namespace containing the Ingress.
  • Two ingresses in different namespaces must not have the same hostname.

Prerequisites

This tutorial requires Kubernetes 1.9 or later. To run the tutorial locally, we recommend using minikube in version v0.28+ with Kubernetes 1.10 (which is the default).

Steps

1. Start Kubernetes recommended Admisson Controllers enabled

To implement admission control rules that validate Kubernetes resources during create, update, and delete operations, you must enable the ValidatingAdmissionWebhook when the Kubernetes API server is started. The ValidatingAdmissionWebhook admission controller is included in the recommended set of admission controllers to enable

Start minikube:

  1. minikube start

Make sure that the minikube ingress addon is enabled:

  1. minikube addons enable ingress

2. Create a new Namespace to deploy OPA into

When OPA is deployed on top of Kubernetes, policies are automatically loaded out of ConfigMaps in the opa namespace.

  1. kubectl create namespace opa

Configure kubectl to use this namespace:

  1. kubectl config set-context opa-tutorial --user minikube --cluster minikube --namespace opa
  2. kubectl config use-context opa-tutorial

3. Deploy OPA on top of Kubernetes

Communication between Kubernetes and OPA must be secured using TLS. To configure TLS, use openssl to create a certificate authority (CA) and certificate/key pair for OPA:

  1. openssl genrsa -out ca.key 2048
  2. openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"

Generate the TLS key and certificate for OPA:

  1. cat >server.conf <<EOF
  2. [req]
  3. req_extensions = v3_req
  4. distinguished_name = req_distinguished_name
  5. [req_distinguished_name]
  6. [ v3_req ]
  7. basicConstraints = CA:FALSE
  8. keyUsage = nonRepudiation, digitalSignature, keyEncipherment
  9. extendedKeyUsage = clientAuth, serverAuth
  10. EOF
  1. openssl genrsa -out server.key 2048
  2. openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
  3. openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

Note: the Common Name value you give to openssl MUST match the name of the OPA service created below.

Create a Secret to store the TLS credentials for OPA:

  1. kubectl create secret tls opa-server --cert=server.crt --key=server.key

Next, use the file below to deploy OPA as an admission controller.

admission-controller.yaml:

  1. # Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
  2. # replicate resources into OPA so they can be used in policies.
  3. kind: ClusterRoleBinding
  4. apiVersion: rbac.authorization.k8s.io/v1
  5. metadata:
  6. name: opa-viewer
  7. roleRef:
  8. kind: ClusterRole
  9. name: view
  10. apiGroup: rbac.authorization.k8s.io
  11. subjects:
  12. - kind: Group
  13. name: system:serviceaccounts:opa
  14. apiGroup: rbac.authorization.k8s.io
  15. ---
  16. # Define role for OPA/kube-mgmt to update configmaps with policy status.
  17. kind: Role
  18. apiVersion: rbac.authorization.k8s.io/v1
  19. metadata:
  20. namespace: opa
  21. name: configmap-modifier
  22. rules:
  23. - apiGroups: [""]
  24. resources: ["configmaps"]
  25. verbs: ["update", "patch"]
  26. ---
  27. # Grant OPA/kube-mgmt role defined above.
  28. kind: RoleBinding
  29. apiVersion: rbac.authorization.k8s.io/v1
  30. metadata:
  31. namespace: opa
  32. name: opa-configmap-modifier
  33. roleRef:
  34. kind: Role
  35. name: configmap-modifier
  36. apiGroup: rbac.authorization.k8s.io
  37. subjects:
  38. - kind: Group
  39. name: system:serviceaccounts:opa
  40. apiGroup: rbac.authorization.k8s.io
  41. ---
  42. kind: Service
  43. apiVersion: v1
  44. metadata:
  45. name: opa
  46. namespace: opa
  47. spec:
  48. selector:
  49. app: opa
  50. ports:
  51. - name: https
  52. protocol: TCP
  53. port: 443
  54. targetPort: 443
  55. ---
  56. apiVersion: extensions/v1beta1
  57. kind: Deployment
  58. metadata:
  59. labels:
  60. app: opa
  61. namespace: opa
  62. name: opa
  63. spec:
  64. replicas: 1
  65. selector:
  66. matchLabels:
  67. app: opa
  68. template:
  69. metadata:
  70. labels:
  71. app: opa
  72. name: opa
  73. spec:
  74. containers:
  75. # WARNING: OPA is NOT running with an authorization policy configured. This
  76. # means that clients can read and write policies in OPA. If you are
  77. # deploying OPA in an insecure environment, be sure to configure
  78. # authentication and authorization on the daemon. See the Security page for
  79. # details: https://www.openpolicyagent.org/docs/security.html.
  80. - name: opa
  81. image: openpolicyagent/opa:0.12.2
  82. args:
  83. - "run"
  84. - "--server"
  85. - "--tls-cert-file=/certs/tls.crt"
  86. - "--tls-private-key-file=/certs/tls.key"
  87. - "--addr=0.0.0.0:443"
  88. - "--addr=http://127.0.0.1:8181"
  89. volumeMounts:
  90. - readOnly: true
  91. mountPath: /certs
  92. name: opa-server
  93. - name: kube-mgmt
  94. image: openpolicyagent/kube-mgmt:0.8
  95. args:
  96. - "--replicate-cluster=v1/namespaces"
  97. - "--replicate=extensions/v1beta1/ingresses"
  98. volumes:
  99. - name: opa-server
  100. secret:
  101. secretName: opa-server
  102. ---
  103. kind: ConfigMap
  104. apiVersion: v1
  105. metadata:
  106. name: opa-default-system-main
  107. namespace: opa
  108. data:
  109. main: |
  110. package system
  111. import data.kubernetes.admission
  112. main = {
  113. "apiVersion": "admission.k8s.io/v1beta1",
  114. "kind": "AdmissionReview",
  115. "response": response,
  116. }
  117. default response = {"allowed": true}
  118. response = {
  119. "allowed": false,
  120. "status": {
  121. "reason": reason,
  122. },
  123. } {
  124. reason = concat(", ", admission.deny)
  125. reason != ""
  126. }
  1. kubectl apply -f admission-controller.yaml

When OPA starts, the kube-mgmt container will load Kubernetes Namespace and Ingress objects into OPA. You can configure the sidecar to load any kind of Kubernetes object into OPA. The sidecar establishes watches on the Kubernetes API server so that OPA has access to an eventually consistent cache of Kubernetes objects.

Next, generate the manifest that will be used to register OPA as an admission controller. This webhook will ignore any namespace with the label openpolicyagent.org/webhook=ignore.

  1. cat > webhook-configuration.yaml <<EOF
  2. kind: ValidatingWebhookConfiguration
  3. apiVersion: admissionregistration.k8s.io/v1beta1
  4. metadata:
  5. name: opa-validating-webhook
  6. webhooks:
  7. - name: validating-webhook.openpolicyagent.org
  8. namespaceSelector:
  9. matchExpressions:
  10. - key: openpolicyagent.org/webhook
  11. operator: NotIn
  12. values:
  13. - ignore
  14. rules:
  15. - operations: ["CREATE", "UPDATE"]
  16. apiGroups: ["*"]
  17. apiVersions: ["*"]
  18. resources: ["*"]
  19. clientConfig:
  20. caBundle: $(cat ca.crt | base64 | tr -d '\n')
  21. service:
  22. namespace: opa
  23. name: opa
  24. EOF

The generated configuration file includes a base64 encoded representation of the CA certificate so that TLS connections can be established between the Kubernetes API server and OPA.

Next label kube-system and the opa namespace so that OPA does not control the resources in those namespaces.

  1. kubectl label ns kube-system openpolicyagent.org/webhook=ignore
  2. kubectl label ns opa openpolicyagent.org/webhook=ignore

Finally, register OPA as an admission controller:

  1. kubectl apply -f webhook-configuration.yaml

You can follow the OPA logs to see the webhook requests being issued by the Kubernetes API server:

  1. kubectl logs -l app=opa -c opa

4. Define a policy and load it into OPA via Kubernetes

To test admission control, create a policy that restricts the hostnames that an ingress can use (ingress-whitelist.rego):

  1. package kubernetes.admission
  2. import data.kubernetes.namespaces
  3. operations = {"CREATE", "UPDATE"}
  4. deny[msg] {
  5. input.request.kind.kind == "Ingress"
  6. operations[input.request.operation]
  7. host := input.request.object.spec.rules[_].host
  8. not fqdn_matches_any(host, valid_ingress_hosts)
  9. msg := sprintf("invalid ingress host %q", [host])
  10. }
  11. valid_ingress_hosts = {host |
  12. whitelist := namespaces[input.request.namespace].metadata.annotations["ingress-whitelist"]
  13. hosts := split(whitelist, ",")
  14. host := hosts[_]
  15. }
  16. fqdn_matches_any(str, patterns) {
  17. fqdn_matches(str, patterns[_])
  18. }
  19. fqdn_matches(str, pattern) {
  20. pattern_parts := split(pattern, ".")
  21. pattern_parts[0] == "*"
  22. str_parts := split(str, ".")
  23. n_pattern_parts := count(pattern_parts)
  24. n_str_parts := count(str_parts)
  25. suffix := trim(pattern, "*.")
  26. endswith(str, suffix)
  27. }
  28. fqdn_matches(str, pattern) {
  29. not contains(pattern, "*")
  30. str == pattern
  31. }

Store the policy in Kubernetes as a ConfigMap. By default kube-mgmt will try to load policies out of configmaps in the opa namespace OR configmaps in other namespaces labelled openpolicyagent.org/policy=rego.

  1. kubectl create configmap ingress-whitelist --from-file=ingress-whitelist.rego

The OPA sidecar will notice the ConfigMap and automatically load the policy into OPA.

5. Exercise the policy

Create two new namespaces to test the Ingress policy.

qa-namespace.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4. annotations:
  5. ingress-whitelist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
  6. name: qa

production-namespace.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4. annotations:
  5. ingress-whitelist: "*.acmecorp.com"
  6. name: production
  1. kubectl create -f qa-namespace.yaml
  2. kubectl create -f production-namespace.yaml

Next, define two Ingress objects. One of the Ingress objects will be permitted and the other will be rejected.

ingress-ok.yaml:

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4. name: ingress-ok
  5. spec:
  6. rules:
  7. - host: signin.acmecorp.com
  8. http:
  9. paths:
  10. - backend:
  11. serviceName: nginx
  12. servicePort: 80

ingress-bad.yaml:

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4. name: ingress-bad
  5. spec:
  6. rules:
  7. - host: acmecorp.com
  8. http:
  9. paths:
  10. - backend:
  11. serviceName: nginx
  12. servicePort: 80

Finally, try to create both Ingress objects:

  1. kubectl create -f ingress-ok.yaml -n production
  2. kubectl create -f ingress-bad.yaml -n qa

The second Ingress is rejected because its hostname does not match the whitelist in the qa namespace.

6. Modify the policy and exercise the changes

OPA allows you to modify policies on-the-fly without recompiling any of the services that offload policy decisions to it.

To enforce the second half of the policy from the start of this tutorial you can load another policy into OPA that prevents Ingress objects in different namespaces from sharing the same hostname.

ingress-conflicts.rego:

  1. package kubernetes.admission
  2. import data.kubernetes.ingresses
  3. deny[msg] {
  4. some other_ns, other_ingress
  5. input.request.kind.kind == "Ingress"
  6. input.request.operation == "CREATE"
  7. host := input.request.object.spec.rules[_].host
  8. ingress := ingresses[other_ns][other_ingress]
  9. other_ns != input.request.namespace
  10. ingress.spec.rules[_].host == host
  11. msg := sprintf("invalid ingress host %q (conflicts with %v/%v)", [host, other_ns, other_ingress])
  12. }
  1. kubectl create configmap ingress-conflicts --from-file=ingress-conflicts.rego

The OPA sidecar annotates ConfigMaps containing policies to indicate if they were installed successfully. Verify that the ConfigMap was installed successfully:

  1. kubectl get configmap ingress-conflicts -o yaml

Test that you cannot create an Ingress in another namespace with the same hostname as the one created earlier.

staging-namespace.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4. annotations:
  5. ingress-whitelist: "*.acmecorp.com"
  6. name: staging
  1. kubectl create -f staging-namespace.yaml
  1. kubectl create -f ingress-ok.yaml -n staging

Wrap Up

Congratulations for finishing the tutorial!

This tutorial showed how you can leverage OPA to enforce admission control decisions in Kubernetes clusters without modifying or recompiling any Kubernetes components. Furthermore, once Kubernetes is configured to use OPA as an External Admission Controller, policies can be modified on-the-fly to satisfy changing operational requirements.

For more information about deploying OPA on top of Kubernetes, see Deployments - Kubernetes.