Getting Per-Route Metrics

To get per-route metrics, you must first create a service profile. Once a service profile has been created, Linkerd will add labels to the Prometheus metrics that associate a specific request to a specific route.

For a tutorial that shows this functionality off, check out the books demo.

You can view per-route metrics in the CLI by running linkerd viz routes:

  1. $ linkerd viz routes svc/webapp
  2. ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
  3. GET / webapp 100.00% 0.6rps 25ms 30ms 30ms
  4. GET /authors/{id} webapp 100.00% 0.6rps 22ms 29ms 30ms
  5. GET /books/{id} webapp 100.00% 1.2rps 18ms 29ms 30ms
  6. POST /authors webapp 100.00% 0.6rps 32ms 46ms 49ms
  7. POST /authors/{id}/delete webapp 100.00% 0.6rps 45ms 87ms 98ms
  8. POST /authors/{id}/edit webapp 0.00% 0.0rps 0ms 0ms 0ms
  9. POST /books webapp 50.76% 2.2rps 26ms 38ms 40ms
  10. POST /books/{id}/delete webapp 100.00% 0.6rps 24ms 29ms 30ms
  11. POST /books/{id}/edit webapp 60.71% 0.9rps 75ms 98ms 100ms
  12. [DEFAULT] webapp 0.00% 0.0rps 0ms 0ms 0ms

The [DEFAULT] route is a catch-all, anything that does not match the regexes specified in your service profile will end up there.

It is also possible to look the metrics up by other resource types, such as:

  1. $ linkerd viz routes deploy/webapp
  2. ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
  3. [DEFAULT] kubernetes 0.00% 0.0rps 0ms 0ms 0ms
  4. GET / webapp 100.00% 0.5rps 27ms 38ms 40ms
  5. GET /authors/{id} webapp 100.00% 0.6rps 18ms 29ms 30ms
  6. GET /books/{id} webapp 100.00% 1.1rps 17ms 28ms 30ms
  7. POST /authors webapp 100.00% 0.5rps 25ms 30ms 30ms
  8. POST /authors/{id}/delete webapp 100.00% 0.5rps 58ms 96ms 99ms
  9. POST /authors/{id}/edit webapp 0.00% 0.0rps 0ms 0ms 0ms
  10. POST /books webapp 45.58% 2.5rps 33ms 82ms 97ms
  11. POST /books/{id}/delete webapp 100.00% 0.6rps 33ms 48ms 50ms
  12. POST /books/{id}/edit webapp 55.36% 0.9rps 79ms 160ms 192ms
  13. [DEFAULT] webapp 0.00% 0.0rps 0ms 0ms 0ms

Then, it is possible to filter all the way down to requests going from a specific resource to other services:

  1. $ linkerd viz routes deploy/webapp --to svc/books
  2. ROUTE SERVICE SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99
  3. DELETE /books/{id}.json books 100.00% 0.5rps 18ms 29ms 30ms
  4. GET /books.json books 100.00% 1.1rps 7ms 12ms 18ms
  5. GET /books/{id}.json books 100.00% 2.5rps 6ms 10ms 10ms
  6. POST /books.json books 52.24% 2.2rps 23ms 34ms 39ms
  7. PUT /books/{id}.json books 41.98% 1.4rps 73ms 97ms 99ms
  8. [DEFAULT] books 0.00% 0.0rps 0ms 0ms 0ms

Troubleshooting

If you’re not seeing any metrics, there are two likely culprits. In both cases, linkerd viz tap can be used to understand the problem. For the resource that the service points to, run:

  1. linkerd viz tap deploy/webapp -o wide | grep req

A sample output is:

  1. req id=3:1 proxy=in src=10.4.0.14:58562 dst=10.4.1.4:7000 tls=disabled :method=POST :authority=webapp:7000 :path=/books/24783/edit src_res=deploy/traffic src_ns=default dst_res=deploy/webapp dst_ns=default rt_route=POST /books/{id}/edit

This will select only the requests observed and show the :authority and rt_route that was used for each request.

  • Linkerd discovers the right service profile to use via :authority or Host headers. The name of your service profile must match these headers. There are many reasons why these would not match, see ingress for one reason. Another would be clients that use IPs directly such as Prometheus.
  • Getting regexes to match can be tough and the ordering is important. Pay attention to rt_route. If it is missing entirely, compare the :path to the regex you’d like for it to match, and use a tester with the Golang flavor of regex.