execute-test

Whether you’re using the included OpenSearch Benchmark workloads or a custom workload, use the execute-test command to gather data about the performance of your OpenSearch cluster according to the selected workload.

Usage

The following example executes a test using the geonames workload in test mode:

  1. opensearch-benchmark execute-test --workload=geonames --test-mode

After the test runs, OpenSearch Benchmark responds with a summary of the benchmark metrics:

  1. ------------------------------------------------------
  2.     _______             __   _____
  3.    / ____(_)___  ____ _/ /  / ___/_________  ________
  4.   / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
  5.  / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
  6. /_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
  7. ------------------------------------------------------
  9. |                         Metric |                 Task |     Value |   Unit |
  10. |-------------------------------:|---------------------:|----------:|-------:|
  11. |            Total indexing time |                      |   28.0997 |    min |
  12. |               Total merge time |                      |   6.84378 |    min |
  13. |             Total refresh time |                      |   3.06045 |    min |
  14. |               Total flush time |                      |  0.106517 |    min |
  15. |      Total merge throttle time |                      |   1.28193 |    min |
  16. |               Median CPU usage |                      |     471.6 |      % |
  17. |             Total Young Gen GC |                      |    16.237 |      s |
  18. |               Total Old Gen GC |                      |     1.796 |      s |
  19. |                     Index size |                      |   2.60124 |     GB |
  20. |                  Total written |                      |   11.8144 |     GB |
  21. |         Heap used for segments |                      |   14.7326 |     MB |
  22. |       Heap used for doc values |                      |  0.115917 |     MB |
  23. |            Heap used for terms |                      |   13.3203 |     MB |
  24. |            Heap used for norms |                      | 0.0734253 |     MB |
  25. |           Heap used for points |                      |    0.5793 |     MB |
  26. |    Heap used for stored fields |                      |  0.643608 |     MB |
  27. |                  Segment count |                      |        97 |        |
  28. |                 Min Throughput |         index-append |   31925.2 | docs/s |
  29. |              Median Throughput |         index-append |   39137.5 | docs/s |
  30. |                 Max Throughput |         index-append |   39633.6 | docs/s |
  31. |      50.0th percentile latency |         index-append |   872.513 |     ms |
  32. |      90.0th percentile latency |         index-append |   1457.13 |     ms |
  33. |      99.0th percentile latency |         index-append |   1874.89 |     ms |
  34. |       100th percentile latency |         index-append |   2711.71 |     ms |
  35. | 50.0th percentile service time |         index-append |   872.513 |     ms |
  36. | 90.0th percentile service time |         index-append |   1457.13 |     ms |
  37. | 99.0th percentile service time |         index-append |   1874.89 |     ms |
  38. |  100th percentile service time |         index-append |   2711.71 |     ms |
  39. |                           ...  |                  ... |       ... |    ... |
  40. |                           ...  |                  ... |       ... |    ... |
  41. |                 Min Throughput |     painless_dynamic |   2.53292 |  ops/s |
  42. |              Median Throughput |     painless_dynamic |   2.53813 |  ops/s |
  43. |                 Max Throughput |     painless_dynamic |   2.54401 |  ops/s |
  44. |      50.0th percentile latency |     painless_dynamic |    172208 |     ms |
  45. |      90.0th percentile latency |     painless_dynamic |    310401 |     ms |
  46. |      99.0th percentile latency |     painless_dynamic |    341341 |     ms |
  47. |      99.9th percentile latency |     painless_dynamic |    344404 |     ms |
  48. |       100th percentile latency |     painless_dynamic |    344754 |     ms |
  49. | 50.0th percentile service time |     painless_dynamic |    393.02 |     ms |
  50. | 90.0th percentile service time |     painless_dynamic |   407.579 |     ms |
  51. | 99.0th percentile service time |     painless_dynamic |   430.806 |     ms |
  52. | 99.9th percentile service time |     painless_dynamic |   457.352 |     ms |
  53. |  100th percentile service time |     painless_dynamic |   459.474 |     ms |
  55. ----------------------------------
  56. [INFO] SUCCESS (took 2634 seconds)
  57. ----------------------------------

Options

Use the following options to customize the execute-test command for your use case. Options in this section are categorized by their use case.

General settings

The following options shape how each test runs and how results appear:

  • --test-mode: Runs the given workload in test mode, which is useful when checking a workload for errors.
  • --user-tag: Defines user-specific key-value pairs to be used in metric record as meta information, for example, intention:baseline-ticket-12345.
  • --results-format: Defines the output format for the command line results, either markdown or csv. Default is markdown.
  • --results-number-align: Defines the column number alignment for when the compare command outputs results. Default is right.
  • --results-file: When provided a file path, writes the compare results to the file indicated in the path.
  • --show-in-results: Determines whether or not to include the comparison in the results file.

Distributions

The following options set which version of OpenSearch and the OpenSearch plugins the benchmark test uses:

  • --distribution-version: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see Version history.
  • --distribution-repository: Defines the repository from where the OpenSearch distribution should be downloaded. Default is release.
  • --revision: Defines the current source code revision to use for running a benchmark test. Default is current.
    • current: Uses the source tree’s current revision based on your OpenSearch distribution.
    • latest: Fetches the latest revision from the main branch of the source tree.
    • You can also use a timestamp or commit ID from the source tree. When using a timestamp, specify @ts, where “ts” is a valid ISO 8601 timestamp, for example, @2013-07-27T10:37:00Z.
  • --opensearch-plugins: Defines which OpenSearch plugins to install. By default, no plugins are installed.
  • --plugin-params: Defines a comma-separated list of key:value pairs that are injected verbatim into all plugins as variables.
  • --runtime-jdk: The major version of JDK to use.
  • --client-options: Defines a comma-separated list of clients to use. All options are passed to the OpenSearch Python client. Default is timeout:60.

Cluster

The following option relates to the target cluster of the benchmark.

  • --target-hosts: Defines a comma-separated list of host-port pairs that should be targeted if using the pipeline benchmark-only. Default is localhost:9200.

Distributed workload generation

The following options help those who want to use multiple hosts to generate load to the benchmark cluster:

  • --load-worker-coordinator-hosts: Defines a comma-separated list of hosts that coordinate loads. Default is localhost.
  • --enable-worker-coordinator-profiling: Enables an analysis of the performance of OpenSearch Benchmark’s worker coordinator. Default is false.

Provisioning

The following options help customize how OpenSearch Benchmark provisions OpenSearch and workloads:

  • --provision-config-repository: Defines the repository from which OpenSearch Benchmark loads provision-configs and provision-config-instances.
  • --provision-config-path: Defines the path to the --provision-config-instance and any OpenSearch plugin configurations to use.
  • --provision-config-revision: Defines a specific Git revision in the provision-config that OpenSearch Benchmark should use.
  • --provision-config-instance: Defines the --provision-config-instance to use. You can see possible configuration instances using the command opensearch-benchmark list provision-config-instances.
  • --provision-config-instance-params: A comma-separated list of key-value pairs injected verbatim as variables for the provision-config-instance.

Workload

The following options determine which workload is used to run the test:

  • --workload-repository: Defines the repository from which OpenSearch Benchmark loads workloads.
  • --workload-path: Defines the path to a downloaded or custom workload.
  • --workload-revision: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use.
  • --workload: Defines the workload to use based on the workload’s name. You can find a list of preloaded workloads using opensearch-benchmark list workloads.

Test procedures

The following options define what test procedures the test uses and which operations are contained inside the procedure:

  • --test-execution-id: Defines a unique ID for this test run.
  • --test-procedure: Defines a test procedure to use. You can find a list of test procedures using opensearch-benchmark list test-procedures.
  • --include-tasks: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
  • --exclude-tasks: Defines a comma-separated list of test procedure tasks not to run.
  • --enable-assertions: Enables assertion checks for tasks. Default is false.

Pipelines

The --pipeline option selects a pipeline to run. You can find a list of pipelines supported by OpenSearch Benchmark by running opensearch-benchmark list pipelines.

Telemetry

The following options enable telemetry devices on OpenSearch Benchmark:

  • --telemetry: Enables the provided telemetry devices when the devices are provided using a comma-separated list. You can find a list of possible telemetry devices by using opensearch-benchmark list telemetry.
  • --telemetry-params: Defines a comma-separated list of key-value pairs that are injected verbatim into the telemetry devices as parameters.

Errors

The following options set how OpenSearch Benchmark handles errors when running tests:

  • --on-error: Controls how OpenSearch Benchmark responds to errors. Default is continue.
    • continue: Continues to run the test despite the error.
    • abort: Aborts the test when an error occurs.
  • --preserve-install: Keeps the Benchmark candidate and its index. Default is false.
  • --kill-running-processes: When set to true, stops any OpenSearch Benchmark processes currently running and allows OpenSearch Benchmark to continue to run. Default is false.