Health Check

This policy enables Kuma to keep track of the health of every data plane proxy, with the goal of minimizing the number of failed requests in case a data plane proxy is temporarily unhealthy.

By creating an HealthCheck resource we can instruct a data plane proxy to keep track of the health status for any other data plane proxy. When health-checks are properly configured, a data plane proxy will never send a request to another data plane proxy that is considered unhealthy. When an unhealthy proxy returns to a healthy state, Kuma will resume sending requests to it again.

This policy provides active checks. If you want to configure passive checks, please utilize the Circuit Breaker policy. Data plane proxies with active checks will explicitly send requests to other data plane proxies to determine if target proxies are healthy or not. This mode generates extra traffic to other proxies and services as described in the policy configuration.

Usage

As usual, we can apply sources and destinations selectors to determine how health-checks will be performed across our data plane proxies.

The HealthCheck policy supports both L4/TCP (default) and L7/HTTP checks.

Examples

TCP

  1. apiVersion: kuma.io/v1alpha1
  2. kind: HealthCheck
  3. mesh: default
  4. metadata:
  5. name: web-to-backend-check
  6. spec:
  7. sources:
  8. - match:
  9. kuma.io/service: web
  10. destinations:
  11. - match:
  12. kuma.io/service: backend
  13. conf:
  14. interval: 10s
  15. timeout: 2s
  16. unhealthyThreshold: 3
  17. healthyThreshold: 1
  18. initialJitter: 5s # optional
  19. intervalJitter: 6s # optional
  20. intervalJitterPercent: 10 # optional
  21. healthyPanicThreshold: 60 # optional, by default 50
  22. failTrafficOnPanic: true # optional, by default false
  23. noTrafficInterval: 10s # optional, by default 60s
  24. eventLogPath: "/tmp/health-check.log" # optional
  25. alwaysLogHealthCheckFailures: true # optional, by default false
  26. tcp:
  27. send: Zm9v
  28. receive:
  29. - YmFy
  30. - YmF6

HTTP

  1. apiVersion: kuma.io/v1alpha1
  2. kind: HealthCheck
  3. mesh: default
  4. metadata:
  5. name: web-to-backend-check
  6. spec:
  7. sources:
  8. - match:
  9. kuma.io/service: web
  10. destinations:
  11. - match:
  12. kuma.io/service: backend
  13. conf:
  14. interval: 10s
  15. timeout: 2s
  16. unhealthyThreshold: 3
  17. healthyThreshold: 1
  18. initialJitter: 5s # optional
  19. intervalJitter: 6s # optional
  20. intervalJitterPercent: 10 # optional
  21. healthyPanicThreshold: 60 # optional, by default 50
  22. failTrafficOnPanic: true # optional, by default false
  23. noTrafficInterval: 10s # optional, by default 60s
  24. eventLogPath: "/tmp/health-check.log" # optional
  25. alwaysLogHealthCheckFailures: true # optional, by default false
  26. http:
  27. path: /health
  28. requestHeadersToAdd:
  29. - append: false
  30. header:
  31. key: Content-Type
  32. value: application/json
  33. - header:
  34. key: Accept
  35. value: application/json
  36. expectedStatuses: [200, 201]

We will apply the configuration with kubectl apply -f [..].

TCP

  1. type: HealthCheck
  2. name: web-to-backend-check
  3. mesh: default
  4. sources:
  5. - match:
  6. kuma.io/service: web
  7. destinations:
  8. - match:
  9. kuma.io/service: backend
  10. conf:
  11. interval: 10s
  12. timeout: 2s
  13. unhealthyThreshold: 3
  14. healthyThreshold: 1
  15. initialJitter: 5s # optional
  16. intervalJitter: 6s # optional
  17. intervalJitterPercent: 10 # optional
  18. healthyPanicThreshold: 60 # optional, by default 50
  19. failTrafficOnPanic: true # optional, by default false
  20. noTrafficInterval: 10s # optional, by default 60s
  21. eventLogPath: "/tmp/health-check.log" # optional
  22. alwaysLogHealthCheckFailures: true # optional, by default false
  23. tcp:
  24. send: Zm9v
  25. receive:
  26. - YmFy
  27. - YmF6

HTTP

  1. type: HealthCheck
  2. name: web-to-backend-check
  3. mesh: default
  4. sources:
  5. - match:
  6. kuma.io/service: web
  7. destinations:
  8. - match:
  9. kuma.io/service: backend
  10. conf:
  11. interval: 10s
  12. timeout: 2s
  13. unhealthyThreshold: 3
  14. healthyThreshold: 1
  15. initialJitter: 5s # optional
  16. intervalJitter: 6s # optional
  17. intervalJitterPercent: 10 # optional
  18. healthyPanicThreshold: 60 # optional, by default 50
  19. failTrafficOnPanic: true # optional, by default false
  20. noTrafficInterval: 10s # optional, by default 60s
  21. eventLogPath: "/tmp/health-check.log" # optional
  22. alwaysLogHealthCheckFailures: true # optional, by default false
  23. http:
  24. path: /health
  25. requestHeadersToAdd:
  26. - append: false
  27. header:
  28. key: Content-Type
  29. value: application/json
  30. - header:
  31. key: Accept
  32. value: application/json
  33. expectedStatuses: [200, 201]

We will apply the configuration with kumactl apply -f [..] or via the HTTP API.

HTTP

HTTP health checks are executed using HTTP 2

  • path - HTTP path which will be requested during the health checks
  • expectedStatuses (optional) - list of status codes which should be considered as a healthy during the checks
    • only statuses in the range [100, 600) are allowed
    • by default, when this property is not provided only responses with status code 200 are being considered healthy
  • requestHeadersToAdd (optional) - list of headers which should be added to every health check request:
    • append (default, optional) - should the value of the provided header be appended to already existing headers (if present)
    • header:
      • key - the name of the header
      • value (optional) - the value of the header

TCP

  • send - Base64 encoded content of the message which should be sent during the health checks
  • receive list of Base64 encoded blocks of strings which should be found in the returning message which should be considered as healthy
    • when checking the response, “fuzzy” matching is performed such that each block must be found, and in the order specified, but not necessarily contiguous;
    • if receive section won’t be provided or will be empty, checks will be performed as “connect only” and will be marked as successful when TCP connection will be successfully established.