This guide demonstrates how to configure rate limiting for HTTP requests destined to a target host that is a part of an OSM managed service mesh.

Prerequisites

  • Kubernetes cluster running Kubernetes v1.22.9 or greater.
  • Have OSM installed.
  • Have kubectl available to interact with the API server.
  • Have osm CLI available for managing the service mesh.
  • OSM version >= v1.2.0.

Demo

The following demo shows a client sending HTTP requests to the fortio service. We will see the impact of applying local HTTP rate limiting policies targeting the fortio service to control the throughput of requests destined to the service backend.

  1. For simplicity, enable permissive traffic policy mode so that explicit SMI traffic access policies are not required for application connectivity within the mesh.

    1. export osm_namespace=osm-system # Replace osm-system with the namespace where OSM is installed
    2. kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}'  --type=merge

  2. Deploy the fortio HTTP service in the demo namespace after enrolling its namespace to the mesh. The fortio HTTP service runs on port 8080.

    1. # Create the demo namespace
    2. kubectl create namespace demo
    4. # Add the namespace to the mesh
    5. osm namespace add demo
    7. # Deploy fortio TCP echo in the demo namespace
    8. kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.2/manifests/samples/fortio/fortio.yaml -n demo

    Confirm the fortio service pod is up and running.

    1. $ kubectl get pods -n demo
    2. NAME                            READY   STATUS    RESTARTS   AGE
    3. fortio-c4bd7857f-7mm6w          2/2     Running   0          22m

  3. Deploy the fortio-client app in the demo namespace. We will use this client to send TCP traffic to the fortio TCP echo service deployed previously.

    1. kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.2/manifests/samples/fortio/fortio-client.yaml -n demo

    Confirm the fortio-client pod is up and running.

    1. $ kubectl get pods -n demo
    2. NAME                            READY   STATUS    RESTARTS   AGE
    3. fortio-client-b9b7bbfb8-prq7r   2/2     Running   0          7s

  4. Confirm the fortio-client app is able to successfully make HTTP requests to the fortio HTTP service on port 8080. We call the fortio service with 3 concurrent connections (-c 3) and send 10 requests (-n 10).

    1. $ fortio_client="$(kubectl get pod -n demo -l app=fortio-client -o jsonpath='{.items[0].metadata.name}')"
    3. $ kubectl exec "$fortio_client" -n demo -c fortio-client -- fortio load -c 3 -n 10 http://fortio.demo.svc.cluster.local:8080
    4. Fortio 1.33.0 running at 8 queries per second, 8->8 procs, for 10 calls: http://fortio.demo.svc.cluster.local:8080
    5. 20:58:07 I httprunner.go:93> Starting http test for http://fortio.demo.svc.cluster.local:8080 with 3 threads at 8.0 qps and parallel warmup
    6. Starting at 8 qps with 3 thread(s) [gomax 8] : exactly 10, 3 calls each (total 9 + 1)
    7. 20:58:08 I periodic.go:723> T002 ended after 1.1273523s : 3 calls. qps=2.661102478790348
    8. 20:58:08 I periodic.go:723> T001 ended after 1.1273756s : 3 calls. qps=2.661047480537986
    9. 20:58:08 I periodic.go:723> T000 ended after 1.5023464s : 4 calls. qps=2.662501803844972
    10. Ended after 1.5024079s : 10 calls. qps=6.656
    11. Sleep times : count 7 avg 0.52874391 +/- 0.03031 min 0.4865562 max 0.5604152 sum 3.7012074
    12. Aggregated Function Time : count 10 avg 0.0050187 +/- 0.005515 min 0.0012575 max 0.0135401 sum 0.050187
    13. # range, mid point, percentile, count
    14. >= 0.0012575 <= 0.002 , 0.00162875 , 70.00, 7
    15. > 0.012 <= 0.0135401 , 0.01277 , 100.00, 3
    16. # target 50% 0.0017525
    17. # target 75% 0.0122567
    18. # target 90% 0.0130267
    19. # target 99% 0.0134888
    20. # target 99.9% 0.013535
    21. Error cases : no data
    22. 20:58:08 I httprunner.go:190> [0] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    23. 20:58:08 I httprunner.go:190> [1] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    24. 20:58:08 I httprunner.go:190> [2] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    25. Sockets used: 3 (for perfect keepalive, would be 3)
    26. Uniform: false, Jitter: false
    27. IP addresses distribution:
    28. 10.96.189.159:8080: 3
    29. Code 200 : 10 (100.0 %)
    30. Response Header Sizes : count 10 avg 124.3 +/- 0.4583 min 124 max 125 sum 1243
    31. Response Body/Total Sizes : count 10 avg 124.3 +/- 0.4583 min 124 max 125 sum 1243
    32. All done 10 calls (plus 0 warmup) 5.019 ms avg, 6.7 qps

    As seen above, all the HTTP requests from the fortio-client pod succeeded.

    1. Code 200 : 10 (100.0 %)

  5. Next, apply a local rate limiting policy to rate limit HTTP requests at the virtual host level to 3 requests per minute.

    1. kubectl apply -f - <<EOF
    2. apiVersion: policy.openservicemesh.io/v1alpha1
    3. kind: UpstreamTrafficSetting
    4. metadata:
    5.   name: http-rate-limit
    6.   namespace: demo
    7. spec:
    8.   host: fortio.demo.svc.cluster.local
    9.   rateLimit:
    10.     local:
    11.       http:
    12.         requests: 3
    13.         unit: minute
    14. EOF

    Confirm no HTTP requests have been rate limited yet by examining the stats on the fortio backend pod.

    1. $ fortio_server="$(kubectl get pod -n demo -l app=fortio -o jsonpath='{.items[0].metadata.name}')"
    3. $ osm proxy get stats "$fortio_server" -n demo | grep 'http_local_rate_limiter.http_local_rate_limit.rate_limited'
    4. http_local_rate_limiter.http_local_rate_limit.rate_limited: 0

  6. Confirm HTTP requests are rate limited.

    1. $ kubectl exec "$fortio_client" -n demo -c fortio-client -- fortio load -c 3 -n 10 http://fortio.demo.svc.cluster.local:8080
    2. Fortio 1.33.0 running at 8 queries per second, 8->8 procs, for 10 calls: http://fortio.demo.svc.cluster.local:8080
    3. 21:06:36 I httprunner.go:93> Starting http test for http://fortio.demo.svc.cluster.local:8080 with 3 threads at 8.0 qps and parallel warmup
    4. Starting at 8 qps with 3 thread(s) [gomax 8] : exactly 10, 3 calls each (total 9 + 1)
    5. 21:06:37 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    6. 21:06:37 W http_client.go:838> [1] Non ok http code 429 (HTTP/1.1 429)
    7. 21:06:37 W http_client.go:838> [2] Non ok http code 429 (HTTP/1.1 429)
    8. 21:06:37 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    9. 21:06:37 W http_client.go:838> [1] Non ok http code 429 (HTTP/1.1 429)
    10. 21:06:37 I periodic.go:723> T001 ended after 1.1269827s : 3 calls. qps=2.661975201571417
    11. 21:06:37 W http_client.go:838> [2] Non ok http code 429 (HTTP/1.1 429)
    12. 21:06:37 I periodic.go:723> T002 ended after 1.1271942s : 3 calls. qps=2.66147572441377
    13. 21:06:38 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    14. 21:06:38 I periodic.go:723> T000 ended after 1.5021191s : 4 calls. qps=2.662904692444161
    15. Ended after 1.5021609s : 10 calls. qps=6.6571
    16. Sleep times : count 7 avg 0.53138026 +/- 0.03038 min 0.4943128 max 0.5602373 sum 3.7196618
    17. Aggregated Function Time : count 10 avg 0.00318326 +/- 0.002431 min 0.0012651 max 0.0077951 sum 0.0318326
    18. # range, mid point, percentile, count
    19. >= 0.0012651 <= 0.002 , 0.00163255 , 60.00, 6
    20. > 0.002 <= 0.003 , 0.0025 , 70.00, 1
    21. > 0.005 <= 0.006 , 0.0055 , 80.00, 1
    22. > 0.006 <= 0.007 , 0.0065 , 90.00, 1
    23. > 0.007 <= 0.0077951 , 0.00739755 , 100.00, 1
    24. # target 50% 0.00185302
    25. # target 75% 0.0055
    26. # target 90% 0.007
    27. # target 99% 0.00771559
    28. # target 99.9% 0.00778715
    29. Error cases : count 7 avg 0.0016392143 +/- 0.000383 min 0.0012651 max 0.0023951 sum 0.0114745
    30. # range, mid point, percentile, count
    31. >= 0.0012651 <= 0.002 , 0.00163255 , 85.71, 6
    32. > 0.002 <= 0.0023951 , 0.00219755 , 100.00, 1
    33. # target 50% 0.00163255
    34. # target 75% 0.00188977
    35. # target 90% 0.00211853
    36. # target 99% 0.00236744
    37. # target 99.9% 0.00239233
    38. 21:06:38 I httprunner.go:190> [0] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    39. 21:06:38 I httprunner.go:190> [1] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    40. 21:06:38 I httprunner.go:190> [2] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    41. Sockets used: 7 (for perfect keepalive, would be 3)
    42. Uniform: false, Jitter: false
    43. IP addresses distribution:
    44. 10.96.189.159:8080: 3
    45. Code 200 : 3 (30.0 %)
    46. Code 429 : 7 (70.0 %)
    47. Response Header Sizes : count 10 avg 37.2 +/- 56.82 min 0 max 124 sum 372
    48. Response Body/Total Sizes : count 10 avg 166 +/- 27.5 min 124 max 184 sum 1660
    49. All done 10 calls (plus 0 warmup) 3.183 ms avg, 6.7 qps

    As seen above, only 3 out of 10 HTTP requests succeeded, while the remaining 7 requests were rate limited as per the rate limiting policy.

    1. Code 200 : 3 (30.0 %)
    2. Code 429 : 7 (70.0 %)

    Examine the stats to further confirm this.

    1. $ osm proxy get stats "$fortio_server" -n demo | grep 'http_local_rate_limiter.http_local_rate_limit.rate_limited'
    2. http_local_rate_limiter.http_local_rate_limit.rate_limited: 7

  7. Next, let’s update our rate limiting policy to allow a burst of requests. Bursts allow a given number of requests over the baseline rate of 3 requests per minute defined by our rate limiting policy.

    1. kubectl apply -f - <<EOF
    2. apiVersion: policy.openservicemesh.io/v1alpha1
    3. kind: UpstreamTrafficSetting
    4. metadata:
    5.   name: http-rate-limit
    6.   namespace: demo
    7. spec:
    8.   host: fortio.demo.svc.cluster.local
    9.   rateLimit:
    10.     local:
    11.       http:
    12.         requests: 3
    13.         unit: minute
    14.         burst: 10
    15. EOF

  8. Confirm the burst capability allows a burst of requests within a small window of time.

    1. $ kubectl exec "$fortio_client" -n demo -c fortio-client -- fortio load -c 3 -n 10 http://fortio.demo.svc.cluster.local:8080
    2. Fortio 1.33.0 running at 8 queries per second, 8->8 procs, for 10 calls: http://fortio.demo.svc.cluster.local:8080
    3. 21:11:04 I httprunner.go:93> Starting http test for http://fortio.demo.svc.cluster.local:8080 with 3 threads at 8.0 qps and parallel warmup
    4. Starting at 8 qps with 3 thread(s) [gomax 8] : exactly 10, 3 calls each (total 9 + 1)
    5. 21:11:05 I periodic.go:723> T002 ended after 1.127252s : 3 calls. qps=2.6613392568831107
    6. 21:11:05 I periodic.go:723> T001 ended after 1.1273028s : 3 calls. qps=2.661219328116634
    7. 21:11:05 I periodic.go:723> T000 ended after 1.5019947s : 4 calls. qps=2.663125242718899
    8. Ended after 1.5020768s : 10 calls. qps=6.6574
    9. Sleep times : count 7 avg 0.53158916 +/- 0.03008 min 0.4943959 max 0.5600713 sum 3.7211241
    10. Aggregated Function Time : count 10 avg 0.00318637 +/- 0.002356 min 0.0012401 max 0.0073302 sum 0.0318637
    11. # range, mid point, percentile, count
    12. >= 0.0012401 <= 0.002 , 0.00162005 , 60.00, 6
    13. > 0.002 <= 0.003 , 0.0025 , 70.00, 1
    14. > 0.005 <= 0.006 , 0.0055 , 80.00, 1
    15. > 0.007 <= 0.0073302 , 0.0071651 , 100.00, 2
    16. # target 50% 0.00184802
    17. # target 75% 0.0055
    18. # target 90% 0.0071651
    19. # target 99% 0.00731369
    20. # target 99.9% 0.00732855
    21. Error cases : no data
    22. 21:11:05 I httprunner.go:190> [0] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    23. 21:11:05 I httprunner.go:190> [1] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    24. 21:11:05 I httprunner.go:190> [2] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    25. Sockets used: 3 (for perfect keepalive, would be 3)
    26. Uniform: false, Jitter: false
    27. IP addresses distribution:
    28. 10.96.189.159:8080: 3
    29. Code 200 : 10 (100.0 %)
    30. Response Header Sizes : count 10 avg 124 +/- 0 min 124 max 124 sum 1240
    31. Response Body/Total Sizes : count 10 avg 124 +/- 0 min 124 max 124 sum 1240
    32. All done 10 calls (plus 0 warmup) 3.186 ms avg, 6.7 qps

    As seen above, all HTTP requests succeeded as we allowed a burst of 10 requests with our rate limiting policy.

    1. Code 200 : 10 (100.0 %)

    Further, examine the stats to confirm the burst allows additional requests to go through. The number of requests rate limited hasn’t increased since our previous rate limit test before we configured the burst setting.

    1. $ osm proxy get stats "$fortio_server" -n demo | grep 'http_local_rate_limiter.http_local_rate_limit.rate_limited'
    2. http_local_rate_limiter.http_local_rate_limit.rate_limited: 7

  9. Next, let’s configure the rate limting policy for a specific HTTP route allowed on the upstream service.

    Note: Since we are using permissive traffic policy mode in the demo, an HTTP route with a wildcard path regex .* is allowed on the upstream backend, so we will configure a rate limiting policy for this route. However, when using SMI policies in the mesh, paths corresponding to matching allowed SMI HTTP routing rules can be configured.

    1. kubectl apply -f - <<EOF
    2. apiVersion: policy.openservicemesh.io/v1alpha1
    3. kind: UpstreamTrafficSetting
    4. metadata:
    5.   name: http-rate-limit
    6.   namespace: demo
    7. spec:
    8.   host: fortio.demo.svc.cluster.local
    9.   httpRoutes:
    10.     - path: .*
    11.       rateLimit:
    12.         local:
    13.           requests: 3
    14.           unit: minute
    15. EOF

  10. Confirm HTTP requests are rate limited at a per-route level.

    1. $ kubectl exec "$fortio_client" -n demo -c fortio-client -- fortio load -c 3 -n 10 http://fortio.demo.svc.cluster.local:8080
    2. Fortio 1.33.0 running at 8 queries per second, 8->8 procs, for 10 calls: http://fortio.demo.svc.cluster.local:8080
    3. 21:19:34 I httprunner.go:93> Starting http test for http://fortio.demo.svc.cluster.local:8080 with 3 threads at 8.0 qps and parallel warmup
    4. Starting at 8 qps with 3 thread(s) [gomax 8] : exactly 10, 3 calls each (total 9 + 1)
    5. 21:19:35 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    6. 21:19:35 W http_client.go:838> [2] Non ok http code 429 (HTTP/1.1 429)
    7. 21:19:35 W http_client.go:838> [1] Non ok http code 429 (HTTP/1.1 429)
    8. 21:19:35 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    9. 21:19:35 W http_client.go:838> [1] Non ok http code 429 (HTTP/1.1 429)
    10. 21:19:35 W http_client.go:838> [2] Non ok http code 429 (HTTP/1.1 429)
    11. 21:19:35 I periodic.go:723> T001 ended after 1.126703s : 3 calls. qps=2.6626360274180505
    12. 21:19:35 I periodic.go:723> T002 ended after 1.1267472s : 3 calls. qps=2.6625315776245104
    13. 21:19:36 W http_client.go:838> [0] Non ok http code 429 (HTTP/1.1 429)
    14. 21:19:36 I periodic.go:723> T000 ended after 1.5027817s : 4 calls. qps=2.6617305760377574
    15. Ended after 1.5028359s : 10 calls. qps=6.6541
    16. Sleep times : count 7 avg 0.53089959 +/- 0.03079 min 0.4903791 max 0.5604715 sum 3.7162971
    17. Aggregated Function Time : count 10 avg 0.00369734 +/- 0.003165 min 0.0011174 max 0.0095033 sum 0.0369734
    18. # range, mid point, percentile, count
    19. >= 0.0011174 <= 0.002 , 0.0015587 , 60.00, 6
    20. > 0.002 <= 0.003 , 0.0025 , 70.00, 1
    21. > 0.007 <= 0.008 , 0.0075 , 90.00, 2
    22. > 0.009 <= 0.0095033 , 0.00925165 , 100.00, 1
    23. # target 50% 0.00182348
    24. # target 75% 0.00725
    25. # target 90% 0.008
    26. # target 99% 0.00945297
    27. # target 99.9% 0.00949827
    28. Error cases : count 7 avg 0.0016556 +/- 0.0004249 min 0.0011174 max 0.0025594 sum 0.0115892
    29. # range, mid point, percentile, count
    30. >= 0.0011174 <= 0.002 , 0.0015587 , 85.71, 6
    31. > 0.002 <= 0.0025594 , 0.0022797 , 100.00, 1
    32. # target 50% 0.0015587
    33. # target 75% 0.00186761
    34. # target 90% 0.00216782
    35. # target 99% 0.00252024
    36. # target 99.9% 0.00255548
    37. 21:19:36 I httprunner.go:190> [0] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    38. 21:19:36 I httprunner.go:190> [1] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    39. 21:19:36 I httprunner.go:190> [2] fortio.demo.svc.cluster.local:8080 resolved to 10.96.189.159:8080
    40. Sockets used: 7 (for perfect keepalive, would be 3)
    41. Uniform: false, Jitter: false
    42. IP addresses distribution:
    43. 10.96.189.159:8080: 3
    44. Code 200 : 3 (30.0 %)
    45. Code 429 : 7 (70.0 %)
    46. Response Header Sizes : count 10 avg 37.2 +/- 56.82 min 0 max 124 sum 372
    47. Response Body/Total Sizes : count 10 avg 166 +/- 27.5 min 124 max 184 sum 1660
    48. All done 10 calls (plus 0 warmup) 3.697 ms avg, 6.7 qps

    As seen above, only 3 out of 10 HTTP requests succeeded, while the remaining 7 requests were rate limited as per the rate limiting policy.

    1. Code 200 : 3 (30.0 %)
    2. Code 429 : 7 (70.0 %)

    Examine the stats to further confirm this. 7 additional requests have been rate limited after configuring HTTP route level rate limiting since our previous test, indicated by the total of 14 HTTP requests rate limited in the stats.

    1. $ osm proxy get stats "$fortio_server" -n demo | grep 'http_local_rate_limiter.http_local_rate_limit.rate_limited'
    2. http_local_rate_limiter.http_local_rate_limit.rate_limited: 14