Securing Your Service

By default, Linkerd automatically enables mutual Transport Layer Security(mTLS) for most HTTP-based communication betweenmeshed pods, by establishing and authenticating secure, private TLS connectionsbetween Linkerd proxies. Simply add yourservices to Linkerd, and Linkerd will take careof the rest.

Linkerd’s automatic mTLS is done in a way that’s completely transparent tothe application. Of course, sometimes it’s helpful to be able to validatewhether mTLS is in effect!

NoteLinkerd uses Kubernetes Service Accounts to define service identity. Thisrequires that the automountServiceAccountToken feature (on by default) hasnot been disabled on the pods. See the Kubernetes service accountdocumentationfor more.

Validating mTLS with linkerd edges

To validate that mTLS is working, you can view a summary of the HTTPconnections between services that are managed by Linkerd using the linkerdedges command. For example:

  1. linkerd -n linkerd edges deployment

The output will look like:

  1. SRC                  DST                  CLIENT                       SERVER                       MSG
  2. linkerd-controller   linkerd-prometheus   linkerd-controller.linkerd   linkerd-prometheus.linkerd   -
  3. linkerd-web          linkerd-controller   linkerd-web.linkerd          linkerd-controller.linkerd   -

In this example, everything is successfully mTLS’d, and the CLIENT andSERVER columns denote the identities used, in the formservice-account-name.namespace. (See Linkerd’s automatic mTLSdocumentation for more on what these identitiesmean.) If there were a problem automatically upgrading the connection withmTLS, the MSG field would contain the reason why.

Validating mTLS with linkerd tap

Instead of relying on an aggregate, it is also possible to watch the requestsand responses in real time to understand what is getting mTLS’d. We can use thelinkerd tap command to sample real time request data.For example:

  1. linkerd -n linkerd tap deploy

Looking at the control plane specifically, there will be two main types of output.

  1. req id=0:7 proxy=in  src=10.138.15.206:51558 dst=10.4.0.18:9998 tls=not_provided_by_remote :method=GET :authority=10.4.0.18:9998 :path=/ready
  2. rsp id=0:7 proxy=in  src=10.138.15.206:51558 dst=10.4.0.18:9998 tls=not_provided_by_remote :status=200 latency=482µs
  3. end id=0:7 proxy=in  src=10.138.15.206:51558 dst=10.4.0.18:9998 tls=not_provided_by_remote duration=32µs response-length=3B

These are calls by the Kubernetes readinessprobe.As probes are initiated from the kubelet, which is not in the mesh, there is noidentity and these requests are not mTLS’d, as denoted by thetls=not_provided_by_remote message.

Other requests to the control plane are TLS’d:

  1. req id=0:11 proxy=in  src=10.4.0.15:54740 dst=10.4.0.17:9090 tls=true :method=GET :authority=linkerd-prometheus.linkerd.svc.cluster.local:9090 :path=/api/v1/query
  2. rsp id=0:11 proxy=in  src=10.4.0.15:54740 dst=10.4.0.17:9090 tls=true :status=200 latency=194886µs
  3. end id=0:11 proxy=in  src=10.4.0.15:54740 dst=10.4.0.17:9090 tls=true duration=121µs response-length=375B

As both linkerd-prometheus and linkerd-web are in the mesh and using HTTPto communicate, the requests are automatically mTLS’d, as denoted by thetls=true output.

Validating mTLS with tshark

The final way to validate mTLS is to look at raw network traffic within thecluster.

Linkerd includes a debug sidecar thatcomes with a selection of commands that make it easier to verify and debug theservice mesh itself. For example, with our emojivoto demoapplication, we can add the debug sidecar by running:

  1. curl -sL https://run.linkerd.io/emojivoto.yml \
  2.   | linkerd inject --enable-debug-sidecar - \
  3.   | kubectl apply -f -

We can then establish a remote shell directly in the debug container of a pod inthe voting service with:

  1. kubectl -n emojivoto exec -it \
  2.     $(kubectl -n emojivoto get po -o name | grep voting) \
  3.     -c linkerd-debug -- /bin/bash

Once we’re inside the debug sidecar, the built-in tshark command can be usedto inspect the raw packets on the network interface. For example:

  1. tshark -i any -d tcp.port==8080,ssl | grep -v 127.0.0.1

This tells tshark that port 8080 might be TLS’d, and to ignore localhost (asthat traffic will always be unencrypted). The output will show both unencryptedcommunication, such as Prometheus scraping metrics, as well as the primaryapplication traffic being automatically mTLS’d.

  1.   131 11.390338699    10.4.0.17 → 10.4.0.23    HTTP 346 GET /metrics HTTP/1.1
  2.   132 11.391486903    10.4.0.23 → 10.4.0.17    HTTP 2039 HTTP/1.1 200 OK  (text/plain)
  3.   133 11.391540872    10.4.0.17 → 10.4.0.23    TCP 68 46766 → 4191 [ACK] Seq=557 Ack=3942 Win=1329 Len=0 TSval=3389590636 TSecr=1915605020
  4.   134 12.128190076    10.4.0.25 → 10.4.0.23    TLSv1.2 154 Application Data
  5.   140 12.129497053    10.4.0.23 → 10.4.0.25    TLSv1.2 149 Application Data
  6.   141 12.129534848    10.4.0.25 → 10.4.0.23    TCP 68 48138 → 8080 [ACK] Seq=1089 Ack=985 Win=236 Len=0 TSval=2234109459 TSecr=617799816
  7.   143 13.140288400    10.4.0.25 → 10.4.0.23    TLSv1.2 150 Application Data
  8.   148 13.141219945    10.4.0.23 → 10.4.0.25    TLSv1.2 136 Application Data

Summary

In this guide, we’ve provided several different ways to validate whetherLinkerd has been able to automatically upgrade connections to mTLS. Note thatthere are several reasons why Linkerd may not be able to do this upgrade—seethe “Caveats and future work” section of the Linkerd automatic mTLSdocumentation—so if you are relying on Linkerdfor security purposes, this kind of validation can be instructive.