This page provides some guidance and best practices around benchmarking the performance of the OPA-Envoy plugin in order to give users an idea of the overhead of using the plugin. It describes an example setup to perform the benchmarks, different benchmarking scenarios and important metrics that should be captured to understand the impact of the OPA-Envoy plugin.

Benchmark Setup

Sample App

The first component of the setup features a simple Go app which provides information about employees in a company. It exposes a /people endpoint to get and create employees. The app’s source code can be found here.

Envoy

Next, is the Envoy proxy that runs alongside the example application. The Envoy configuration below defines an external authorization filter envoy.ext_authz for a gRPC authorization server. The config uses Envoy’s in-built gRPC client which is a minimal custom implementation of gRPC to make the external gRPC call.

  1. static_resources:
  2.   listeners:
  3.   - address:
  4.       socket_address:
  5.         address: 0.0.0.0
  6.         port_value: 8000
  7.     filter_chains:
  8.     - filters:
  9.       - name: envoy.http_connection_manager
  10.         typed_config:
  11.           "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
  12.           codec_type: auto
  13.           stat_prefix: ingress_http
  14.           route_config:
  15.             name: local_route
  16.             virtual_hosts:
  17.             - name: backend
  18.               domains:
  19.               - "*"
  20.               routes:
  21.               - match:
  22.                   prefix: "/"
  23.                 route:
  24.                   cluster: service
  25.           http_filters:
  26.           - name: envoy.ext_authz
  27.             typed_config:
  28.               "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
  29.               transport_api_version: V3
  30.               with_request_body:
  31.                 max_request_bytes: 8192
  32.                 allow_partial_message: true
  33.               failure_mode_allow: false
  34.               grpc_service:
  35.                 envoy_grpc:
  36.                   cluster_name: opa-envoy
  37.                 timeout: 0.5s
  38.           - name: envoy.filters.http.router
  39.   clusters:
  40.   - name: service
  41.     connect_timeout: 0.25s
  42.     type: strict_dns
  43.     lb_policy: round_robin
  44.     load_assignment:
  45.       cluster_name: service
  46.       endpoints:
  47.       - lb_endpoints:
  48.         - endpoint:
  49.             address:
  50.               socket_address:
  51.                 address: 127.0.0.1
  52.                 port_value: 8080
  53.   - name: opa-envoy
  54.     connect_timeout: 1.25s
  55.     type: strict_dns
  56.     lb_policy: round_robin
  57.     http2_protocol_options: {}
  58.     load_assignment:
  59.       cluster_name: opa-envoy
  60.       endpoints:
  61.       - lb_endpoints:
  62.         - endpoint:
  63.             address:
  64.               socket_address:
  65.                 address: 127.0.0.1
  66.                 port_value: 9191
  67. admin:
  68.   access_log_path: "/dev/null"
  69.   address:
  70.     socket_address:
  71.       address: 0.0.0.0
  72.       port_value: 8001
  73. layered_runtime:
  74.   layers:
  75.     - name: static_layer_0
  76.       static_layer:
  77.         envoy:
  78.           resource_limits:
  79.             listener:
  80.               example_listener_name:
  81.                 connection_limit: 10000
  82.         overload:
  83.           global_downstream_max_connections: 50000

OPA-Envoy Plugin

Now let’s deploy OPA as an External Authorization server. Below is a sample configuration for the OPA-Envoy container:

  1. containers:
  2. - image: openpolicyagent/opa:0.31.0-envoy
  3.   imagePullPolicy: IfNotPresent
  4.   name: opa
  5.   resources:
  6.     requests:
  7.       memory: "64Mi"
  8.       cpu: "1m"
  9.     limits:
  10.       memory: "128Mi"
  11.       cpu: "2m"
  12.   args:
  13.   - "run"
  14.   - "--server"
  15.   - "--addr=localhost:8181"
  16.   - "--diagnostic-addr=0.0.0.0:8282"
  17.   - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
  18.   - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
  19.   - "--ignore=.*"
  20.   - "/policy/policy.rego"
  21.   livenessProbe:
  22.     httpGet:
  23.       path: /health?plugins
  24.       port: 8282
  25.   readinessProbe:
  26.     httpGet:
  27.       path: /health?plugins
  28.       port: 8282

💡 Consider specifying CPU and memory resource requests and limits for the OPA and other containers to prevent deployments from resource starvation. You can also start OPA with the GOMAXPROCSenvironment variable to limit the number of cores that OPA can consume.

💡 The OPA-Envoy plugin can be configured to listen on a UNIX Domain Socket. A complete example of such a setup can be found here.

Load Generator And Measurement Tool

Consider using a load generator and measurement tool that measures latency from the end user’s perspective and reports latency as the percentiles of a distribution, e.g. p50 (median), p99, p999 etc. As example implementation of such a tool can be found here.

Benchmark Scenarios

Following are some scenarios to perform benchmarks on. The results could be used to compare OPA-Envoy plugin’s latency and resource consumption with the baseline (no-opa) case for instance.

  • App Only

In this case, requests are sent directly to the application ie. no Envoy and OPA in the request path.

  • App and Envoy

In this case, OPA is not included in the request path but Envoy is (ie. Envoy External Authorization API disabled).

  • App, Envoy and OPA (NOP policy)

In this case, performance measurements are observed with Envoy External Authorization API enabled. This means Envoy will make a call to OPA on every incoming request with the below NOP policy loaded into OPA.

  1. package envoy.authz
  3. default allow = true
  • App, Envoy and OPA (RBAC policy)

In this case, performance measurements are observed with Envoy External Authorization API enabled and a sample real-world RBAC policy as shown below loaded into OPA.

  1. package envoy.authz
  3. import input.attributes.request.http as http_request
  5. default allow = false
  7. allow {
  8.     roles_for_user[r]
  9.     required_roles[r]
  10. }
  12. roles_for_user[r] {
  13.     r := user_roles[user_name][_]
  14. }
  16. required_roles[r] {
  17.     perm := role_perms[r][_]
  18.     perm.method = http_request.method
  19.     perm.path = http_request.path
  20. }
  22. user_name = parsed {
  23.     [_, encoded] := split(http_request.headers.authorization, " ")
  24.     [parsed, _] := split(base64url.decode(encoded), ":")
  25. }
  27. user_roles = {
  28.     "alice": ["guest"],
  29.     "bob": ["admin"]
  30. }
  32. role_perms = {
  33.     "guest": [
  34.         {"method": "GET",  "path": "/people"},
  35.     ],
  36.     "admin": [
  37.         {"method": "GET",  "path": "/people"},
  38.         {"method": "POST",  "path": "/people"},
  39.     ],
  40. }
  • App, Envoy and OPA (Header Injection policy)

This scenario is similar to the previous one expect the policy decision is an object which contains optional response headers. An example of such a policy can be found here.

Measurements

This section describes some metrics that should help to measure the cost of the OPA-Envoy plugin in terms of CPU and memory consumed as well as latency added.

  • End-to-end Latency is the latency measured from the end user’s perspective. This includes time spent on the network, in the application, in OPA and so on. The sample load tester tool shows how to measure this metric.

  • OPA Evaluation is the time taken to evaluate the policy.

  • gRPC Server Handler is the total time taken to prepare the input for the policy, evaluate the policy (OPA Evaluation) and prepare the result. Basically this is time spent by the OPA-Envoy plugin to process the request. OPA’s metrics package provides helpers to measure both gRPC Server Handler and OPA Evaluation time.

  • Resource utilization refers to the CPU and memory usage of the OPA-Envoy container. kubectl top utility can be leveraged to measure this.

Features

The sample OPA-Envoy deployment described previously, does not utilize OPA’s decision logs management API that enables periodic reporting of decision logs to remote HTTP servers or local console. Decision logging can be enabled by updating the OPA-Envoy configuration, and the guidance provided on this page can be used to gather benchmark results.