This guide will demonstrate how to configure HTTP and HTTPS ingress to a service part of an OSM managed service mesh when using Kubernetes Nginx Ingress Controller.

Prerequisites

  • Kubernetes cluster running Kubernetes v1.20.0 or greater.
  • Have kubectl available to interact with the API server.
  • Have OSM version >= v0.10.0 installed.
  • Have Kubernetes Nginx Ingress Controller installed. Refer to the deployment guide to install it.

Demo

First, note the details regarding OSM and Nginx installations:

  1. osm_namespace=osm-system # Replace osm-system with the namespace where OSM is installed
  2. osm_mesh_name=osm # replace osm with the mesh name (use `osm mesh list` command)
  4. nginx_ingress_namespace=<nginx-namespace> # replace <nginx-namespace> with the namespace where Nginx is installed
  5. nginx_ingress_service=<nginx-ingress-controller-service> # replace <nginx-ingress-controller-service> with the name of the nginx ingress controller service
  6. nginx_ingress_host="$(kubectl -n "$nginx_ingress_namespace" get service "$nginx_ingress_service" -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
  7. nginx_ingress_port="$(kubectl -n "$nginx_ingress_namespace" get service "$nginx_ingress_service" -o jsonpath='{.spec.ports[?(@.name=="http")].port}')"

To restrict ingress traffic on backends to authorized clients, we will set up the IngressBackend configuration such that only ingress traffic from the endpoints of the Nginx Ingress Controller service can route traffic to the service backend. To be able to discover the endpoints of this service, we need OSM controller to monitor the corresponding namespace. However, Nginx must NOT be injected with an Envoy sidecar to function properly.

  1. osm namespace add "$nginx_ingress_namespace" --mesh-name "$osm_mesh_name" --disable-sidecar-injection

Next, we will deploy the sample httpbin service.

  1. # Create a namespace
  2. kubectl create ns httpbin
  4. # Add the namespace to the mesh
  5. osm namespace add httpbin
  7. # Deploy the application
  8. kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/httpbin/httpbin.yaml -n httpbin

Confirm the httpbin service and pod is up and running:

  1. $ kubectl get pods -n httpbin
  2. NAME                       READY   STATUS    RESTARTS   AGE
  3. httpbin-74677b7df7-zzlm2   2/2     Running   0          11h
  5. $ kubectl get svc -n httpbin
  6. NAME      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
  7. httpbin   ClusterIP   10.0.22.196   <none>        14001/TCP   11h

HTTP Ingress

Next, we will create the Ingress and IngressBackend configurations necessary to allow external clients to access the httpbin service on port 14001 in the httpbin namespace. The connection from the Nginx’s ingress service to the httpbin backend pod will be unencrypted since we aren’t using TLS.

  1. kubectl apply -f - <<EOF
  2. apiVersion: networking.k8s.io/v1
  3. kind: Ingress
  4. metadata:
  5.   name: httpbin
  6.   namespace: httpbin
  7. spec:
  8.   ingressClassName: nginx
  9.   rules:
  10.   - http:
  11.       paths:
  12.       - path: /
  13.         pathType: Prefix
  14.         backend:
  15.           service:
  16.             name: httpbin
  17.             port:
  18.               number: 14001
  19. ---
  20. kind: IngressBackend
  21. apiVersion: policy.openservicemesh.io/v1alpha1
  22. metadata:
  23.   name: httpbin
  24.   namespace: httpbin
  25. spec:
  26.   backends:
  27.   - name: httpbin
  28.     port:
  29.       number: 14001 # targetPort of httpbin service
  30.       protocol: http
  31.   sources:
  32.   - kind: Service
  33.     namespace: "$nginx_ingress_namespace"
  34.     name: "$nginx_ingress_service"
  35. EOF

Now, we expect external clients to be able to access the httpbin service for HTTP requests:

  1. $ curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
  2. HTTP/1.1 200 OK
  3. Date: Wed, 18 Aug 2021 18:12:35 GMT
  4. Content-Type: application/json
  5. Content-Length: 366
  6. Connection: keep-alive
  7. access-control-allow-origin: *
  8. access-control-allow-credentials: true
  9. x-envoy-upstream-service-time: 2

HTTPS Ingress (mTLS and TLS)

To proxy connections to HTTPS backends, we will configure the Ingress and IngressBackend configurations to use https as the backend protocol, and have OSM issue a certificate that Nginx will use as the client certificate to proxy HTTPS connections to TLS backends. The client certificate and CA certificate will be stored in a Kubernetes secret that Nginx will use to authenticate service mesh backends.

To issue a client certificate for the Nginx ingress service, update the osm-mesh-config MeshConfig resource.

  1. kubectl edit meshconfig osm-mesh-config -n "$osm_namespace"

Add the ingressGateway field under spec.certificate:

  1. certificate:
  2.   ingressGateway:
  3.     secret:
  4.       name: osm-nginx-client-cert
  5.       namespace: <osm-namespace> # replace <osm-namespace> with the namespace where OSM is installed
  6.     subjectAltNames:
  7.     - ingress-nginx.ingress-nginx.cluster.local
  8.     validityDuration: 24h

Note: The Subject Alternative Name (SAN) is of the form <service-account>.<namespace>.cluster.local, where the service account and namespace correspond to the Ngnix service.

Next, we need to create an Ingress and IngressBackend configuration to use TLS proxying to the backend service, while enabling proxying to the backend over mTLS. For this to work, we must create an IngressBackend resource that specifies HTTPS ingress traffic directed to the httpbin service must only accept traffic from a trusted client. OSM provisioned a client certificate for the Nginx ingress service with the Subject ALternative Name (SAN) ingress-nginx.ingress-nginx.cluster.local, so the IngressBackend configuration needs to reference the same SAN for mTLS authentication between the Nginx ingress service and the httpbin backend.

Apply the configurations:

  1. kubectl apply -f - <<EOF
  2. apiVersion: networking.k8s.io/v1
  3. kind: Ingress
  4. metadata:
  5.   name: httpbin
  6.   namespace: httpbin
  7.   annotations:
  8.     nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  9.     # proxy_ssl_name for a service is of the form <service-account>.<namespace>.cluster.local
  10.     nginx.ingress.kubernetes.io/configuration-snippet: |
  11.       proxy_ssl_name "httpbin.httpbin.cluster.local";
  12.     nginx.ingress.kubernetes.io/proxy-ssl-secret: "osm-system/osm-nginx-client-cert"
  13.     nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
  14. spec:
  15.   ingressClassName: nginx
  16.   rules:
  17.   - http:
  18.       paths:
  19.       - path: /
  20.         pathType: Prefix
  21.         backend:
  22.           service:
  23.             name: httpbin
  24.             port:
  25.               number: 14001
  26. ---
  27. apiVersion: policy.openservicemesh.io/v1alpha1
  28. kind: IngressBackend
  29. metadata:
  30.   name: httpbin
  31.   namespace: httpbin
  32. spec:
  33.   backends:
  34.   - name: httpbin
  35.     port:
  36.       number: 14001 # targetPort of httpbin service
  37.       protocol: https
  38.     tls:
  39.       skipClientCertValidation: false
  40.   sources:
  41.   - kind: Service
  42.     name: "$nginx_ingress_service"
  43.     namespace: "$nginx_ingress_namespace"
  44.   - kind: AuthenticatedPrincipal
  45.     name: ingress-nginx.ingress-nginx.cluster.local
  46. EOF

Now, we expect external clients to be able to access the httpbin service for requests with HTTPS proxying over mTLS between the ingress gateway and service backend:

  1. $ curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
  2. HTTP/1.1 200 OK
  3. Date: Wed, 18 Aug 2021 18:12:35 GMT
  4. Content-Type: application/json
  5. Content-Length: 366
  6. Connection: keep-alive
  7. access-control-allow-origin: *
  8. access-control-allow-credentials: true
  9. x-envoy-upstream-service-time: 2

To verify that unauthorized clients are not allowed to access the backend, update the sources specified in the IngressBackend configuration. Let’s update the principal to something other than the SAN encoded in the Nginx client’s certificate.

  1. kubectl apply -f - <<EOF
  2. apiVersion: policy.openservicemesh.io/v1alpha1
  3. kind: IngressBackend
  4. metadata:
  5.   name: httpbin
  6.   namespace: httpbin
  7. spec:
  8.   backends:
  9.   - name: httpbin
  10.     port:
  11.       number: 14001 # targetPort of httpbin service
  12.       protocol: https
  13.     tls:
  14.       skipClientCertValidation: false
  15.   sources:
  16.   - kind: Service
  17.     name: "$nginx_ingress_service"
  18.     namespace: "$nginx_ingress_namespace"
  19.   - kind: AuthenticatedPrincipal
  20.     name: untrusted-client.cluster.local # untrusted
  21. EOF

Confirm the requests are rejected with an HTTP 403 Forbidden response:

  1. $ curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
  2. HTTP/1.1 403 Forbidden
  3. Date: Wed, 18 Aug 2021 18:36:09 GMT
  4. Content-Type: text/plain
  5. Content-Length: 19
  6. Connection: keep-alive

Next, we demonstrate support for disabling client certificate validation on the service backend if necessary, by updating our IngressBackend configuration to set skipClientCertValidation: true, while still using an untrusted client:

  1. kubectl apply -f - <<EOF
  2. apiVersion: policy.openservicemesh.io/v1alpha1
  3. kind: IngressBackend
  4. metadata:
  5.   name: httpbin
  6.   namespace: httpbin
  7. spec:
  8.   backends:
  9.   - name: httpbin
  10.     port:
  11.       number: 14001 # targetPort of httpbin service
  12.       protocol: https
  13.     tls:
  14.       skipClientCertValidation: true
  15.   sources:
  16.   - kind: Service
  17.     name: "$nginx_ingress_service"
  18.     namespace: "$nginx_ingress_namespace"
  19.   - kind: AuthenticatedPrincipal
  20.     name: untrusted-client.cluster.local # untrusted
  21. EOF

Confirm the requests succeed again since untrusted authenticated principals are allowed to connect to the backend:

  1. $ curl -sI http://"$nginx_ingress_host":"$nginx_ingress_port"/get
  2. HTTP/1.1 200 OK
  3. Date: Wed, 18 Aug 2021 18:36:49 GMT
  4. Content-Type: application/json
  5. Content-Length: 364
  6. Connection: keep-alive
  7. access-control-allow-origin: *
  8. access-control-allow-credentials: true
  9. x-envoy-upstream-service-time: 2