End user RBAC

In this lab we will set up role-based access control (RBAC) suitable for running the cluster in production. We will cover roles for using Calico. General RBAC for a production Kubernetes cluster is beyond the scope of this lab.

Using calicoctl

In order for the calicoctl tool to perform version mismatch verification (to make sure the versions for both the cluster and calicoctl are the same), whoever is using it needs to have get access to clusterinformations at the cluster level, i.e., not in a namespace. The network admin role below already has such access, but we will see that we will need to add it to the service owner user we will create further on.

kubectl apply -f - <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calicoctl-user
rules:
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
EOF

Network admin

A network admin is a person responsible for configuring and operating the Calico network as a whole. As such, they will need access to all Calico custom resources, as well as some associated Kubernetes resources.

Create the role

kubectl apply -f - <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: network-admin
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
    verbs:
      - get
      - watch
      - list
      - update
  - apiGroups: [""]
    resources:
      - namespaces
      - serviceaccounts
    verbs:
      - get
      - watch
      - list
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs: ["*"]
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - felixconfigurations
      - ipamblocks
      - blockaffinities
      - ipamhandles
      - ipamconfigs
      - bgppeers
      - bgpconfigurations
      - ippools
      - hostendpoints
      - clusterinformations
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
    verbs: ["*"]
EOF

To test out the network admin role, we’ll create a user named Nik grant them the role.

On the Kubernetes control plane node, create the key and certificate signing request. Note that we include /O=network-admins in the subject. This places Nik in the network-admins group.

openssl req -newkey rsa:4096 \
           -keyout nik.key \
           -nodes \
           -out nik.csr \
           -subj "/O=network-admins/CN=nik"

We will sign this certificate using the main Kubernetes CA.

sudo openssl x509 -req -in nik.csr \
                  -CA /etc/kubernetes/pki/ca.crt \
                  -CAkey /etc/kubernetes/pki/ca.key \
                  -CAcreateserial \
                  -out nik.crt \
                  -days 365
sudo chown $(id -u):$(id -g) nik.crt

Next, we create a kubeconfig file for Nik.

APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}')
kubectl config set-cluster kubernetes \
    --certificate-authority=/etc/kubernetes/pki/ca.crt \
    --embed-certs=true \
    --server=$APISERVER \
    --kubeconfig=nik.kubeconfig
kubectl config set-credentials nik \
    --client-certificate=nik.crt \
    --client-key=nik.key \
    --embed-certs=true \
    --kubeconfig=nik.kubeconfig
kubectl config set-context default \
    --cluster=kubernetes \
    --user=nik \
    --kubeconfig=nik.kubeconfig
kubectl config use-context default --kubeconfig=nik.kubeconfig

Bind the role to the group network-admins.

kubectl create clusterrolebinding network-admins --clusterrole=network-admin --group=network-admins

Test Nik’s access by creating a global network set

KUBECONFIG=./nik.kubeconfig calicoctl apply -f - <<EOF
apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: niks-set
spec:
  nets:
  - 110.120.130.0/24
  - 210.220.230.0/24
EOF

Verify the global network set exists

KUBECONFIG=./nik.kubeconfig calicoctl get globalnetworkset -o wide

Result

NAME       NETS
niks-set   110.120.130.0/24,210.220.230.0/24

Delete the global network set

KUBECONFIG=./nik.kubeconfig calicoctl delete globalnetworkset niks-set

Service owner

A service owner is a person responsible for operating one or more services in Kubernetes. They should be able to define network policy for their service, but don’t need to view or modify any global configuration related to Calico.

Define the role

kubectl apply -f - <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: network-service-owner
rules:
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs: ["*"]
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - networkpolicies
      - networksets
    verbs: ["*"]
EOF

To test out the service owner role, we’ll create a user named Sam and grant them the role.

On the Kubernetes control plane node, create the key and certificate signing request.

openssl req -newkey rsa:4096 \
           -keyout sam.key \
           -nodes \
           -out sam.csr \
           -subj "/CN=sam"

We will sign this certificate using the main Kubernetes CA.

sudo openssl x509 -req -in sam.csr \
                  -CA /etc/kubernetes/pki/ca.crt \
                  -CAkey /etc/kubernetes/pki/ca.key \
                  -CAcreateserial \
                  -out sam.crt \
                  -days 365
sudo chown $(id -u):$(id -g) sam.crt

Next, we create a kubeconfig file for Sam.

APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}')
kubectl config set-cluster kubernetes \
    --certificate-authority=/etc/kubernetes/pki/ca.crt \
    --embed-certs=true \
    --server=$APISERVER \
    --kubeconfig=sam.kubeconfig
kubectl config set-credentials sam \
    --client-certificate=sam.crt \
    --client-key=sam.key \
    --embed-certs=true \
    --kubeconfig=sam.kubeconfig
kubectl config set-context default \
    --cluster=kubernetes \
    --user=sam \
    --kubeconfig=sam.kubeconfig
kubectl config use-context default --kubeconfig=sam.kubeconfig

We will limit Sam’s access to a single namespace. Create the namespace

kubectl create namespace sam

Bind the role to Sam in the namespace

kubectl create rolebinding -n sam network-service-owner-sam --clusterrole=network-service-owner --user=sam

Also bind the calicoctl-user role to sam at the cluster level so that they can use calicoctl properly

kubectl create clusterrolebinding calicoctl-user-sam --clusterrole=calicoctl-user --user=sam

Sam cannot create global network set resources (like Nik can as network admin)

KUBECONFIG=./sam.kubeconfig calicoctl get globalnetworkset -o wide

Result

connection is unauthorized: globalnetworksets.crd.projectcalico.org is forbidden: User "sam" cannot list resource "globalnetworksets" in API group "crd.projectcalico.org" at the cluster scope

However, Sam can create resources in their own namespace

KUBECONFIG=./sam.kubeconfig calicoctl apply -f - <<EOF
apiVersion: projectcalico.org/v3
kind: NetworkSet
metadata:
  name: sams-set
  namespace: sam
spec:
  nets:
  - 110.120.130.0/24
  - 210.220.230.0/24
EOF

Verify the resource exists

KUBECONFIG=./sam.kubeconfig calicoctl get networksets -n sam

Result

NAMESPACE   NAME
sam         sams-set

Delete the NetworkSet

KUBECONFIG=./sam.kubeconfig calicoctl delete networkset sams-set -n sam

Next

Istio integration