Custom CA Integration using Kubernetes CSR

This feature is actively in development and is considered experimental.

This feature requires Kubernetes version >= 1.18.

This task shows how to provision Workload Certificates using a custom certificate authority that integrates with the Kubernetes CSR API. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk MTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.

For this example, we use open-source cert-manager. Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests starting with version 1.4.

Deploy Custom CA controller in the Kubernetes cluster

  1. Deploy cert-manager according to the installation doc.

    Note: Make sure to enable feature gate: --feature-gates=ExperimentalCertificateSigningRequestControllers=true

  2. Create three self signed cluster issuers istio-system, foo and bar for cert-manager. Note: Namespace issuers and other types of issuers can also be used.

    ` ``` $ cat < ./selfsigned-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-bar-issuer spec:

    selfSigned: {}

    apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: bar-ca namespace: cert-manager spec: isCA: true commonName: bar secretName: bar-ca-selfsigned issuerRef:

    1. name: selfsigned-bar-issuer
    2. kind: ClusterIssuer
    3. group: cert-manager.io

    apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: bar spec: ca:

    1. secretName: bar-ca-selfsigned

    apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-foo-issuer spec:

    selfSigned: {}

    apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: foo-ca namespace: cert-manager spec: isCA: true commonName: foo secretName: foo-ca-selfsigned issuerRef:

    1. name: selfsigned-foo-issuer
    2. kind: ClusterIssuer
    3. group: cert-manager.io

    apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: foo spec: ca:

    1. secretName: foo-ca-selfsigned

    apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-istio-issuer spec:

    selfSigned: {}

    apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: istio-ca namespace: cert-manager spec: isCA: true commonName: istio-system secretName: istio-ca-selfsigned issuerRef:

    1. name: selfsigned-istio-issuer
    2. kind: ClusterIssuer
    3. group: cert-manager.io

    apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: istio-system spec: ca:

    1. secretName: istio-ca-selfsigned

    EOF $ kubectl apply -f ./selfsigned-issuer.yaml ``

Export root certificates for each cluster issuer

` ``` $ export istioca=$(kubectl get clusterissuers istio-system -o jsonpath=’{.spec.ca.secretName}’ | xargs kubectl get secret -n cert-manager -o jsonpath=’{.data.ca.crt}’ | base64 -d)

$ export fooca=$(kubectl get clusterissuers foo -o jsonpath=’{.spec.ca.secretName}’ | xargs kubectl get secret -n cert-manager -o jsonpath=’{.data.ca.crt}’ | base64 -d)

$ export barca=$(kubectl get clusterissuers bar -o jsonpath=’{.spec.ca.secretName}’ | xargs kubectl get secret -n cert-manager -o jsonpath=’{.data.ca.crt}’ | base64 -d) ``

Deploy Istio with default cert-signer info

  1. Deploy Istio on the cluster using istioctl with the following configuration. The ISTIO_META_CERT_SIGNER is the default cert-signer for workloads.

    $ cat <<EOF > ./istio.yaml apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: defaultConfig: proxyMetadata: ISTIO_META_CERT_SIGNER: istio-system caCertificates: - pem: | $istioca certSigners: - clusterissuers.cert-manager.io/istio-system - pem: | $fooca certSigners: - clusterissuers.cert-manager.io/foo - pem: | $barca certSigners: - clusterissuers.cert-manager.io/bar components: pilot: k8s: env: - name: CERT_SIGNER_DOMAIN value: clusterissuers.cert-manager.io - name: EXTERNAL_CA value: ISTIOD_RA_KUBERNETES_API - name: PILOT_CERT_PROVIDER value: k8s.io/clusterissuers.cert-manager.io/istio-system overlays: - kind: ClusterRole name: istiod-clusterrole-istio-system patches: - path: rules[-1] value: | apiGroups: - certificates.k8s.io resourceNames: - clusterissuers.cert-manager.io/foo - clusterissuers.cert-manager.io/bar - clusterissuers.cert-manager.io/istio-system resources: - signers verbs: - approve EOF $ istioctl install -f ./istio.yaml

  2. Deploy the proxyconfig-bar.yaml in the bar namespace to define cert-signer for workloads in the bar namespace.

    $ cat <<EOF > ./proxyconfig-bar.yaml apiVersion: networking.istio.io/v1beta1 kind: ProxyConfig metadata: name: barpc namespace: bar spec: environmentVariables: ISTIO_META_CERT_SIGNER: bar EOF $ kubectl apply -f ./proxyconfig-bar.yaml

  3. Deploy the proxyconfig-foo.yaml in the foo namespace to define cert-signer for workloads in the foo namespace.

    $ cat <<EOF > ./proxyconfig-bar.yaml apiVersion: networking.istio.io/v1beta1 kind: ProxyConfig metadata: name: foopc namespace: foo spec: environmentVariables: ISTIO_META_CERT_SIGNER: foo EOF $ kubectl apply -f ./proxyconfig-foo.yaml

  4. Deploy the httpbin and sleep sample application in the foo and bar namespaces.

    $ kubectl label ns foo istio-injection=enabled $ kubectl label ns bar istio-injection=enabled $ kubectl apply -f samples/httpbin/httpbin.yaml -n foo $ kubectl apply -f samples/sleep/sleep.yaml -n foo $ kubectl apply -f samples/httpbin/httpbin.yaml -n bar $ kubectl apply -f samples/sleep/sleep.yaml -n bar

Verify the network connectivity between httpbin and sleep within the same namespace

When the workloads are deployed, they send CSR Requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer or issuer to sign the cert back. Workloads under foo namespace will use foo cluster issuers while workloads under bar namespace will use the bar cluster issuers. To verify that they have indeed been signed by correct cluster issuers, We can verify workloads under the same namespace can communicate will while workloads under the different namespace cannot communicate.

  1. Check network connectivity between service sleep and httpbin in the foo namespace.

    $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={.items..metadata.name}) $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.foo:8000/html <!DOCTYPE html> <html> <head> </head> <body> <h1>Herman Melville - Moby-Dick</h1> <div> <p> Availing himself of the mild... </p> </div> </body>

  2. Check network connectivity between service sleep in the foo namespace and httpbin in the bar namespace.

    $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={ .items..metadata.name}) $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.bar:8000/html upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Cleanup

  • Remove the istio-system, foo and bar namespaces:

    $ kubectl delete ns istio-system $ kubectl delete ns foo $ kubectl delete ns bar

Reasons to use this feature

  • Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the CertificateSigningRequest Resources and act on them.

  • Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.