Remote Store Stats API

Introduced 2.8

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated GitHub issue.

Use the Remote Store Stats API to monitor shard-level remote store performance.

Path and HTTP methods

  1. GET _remotestore/stats/<index_name>
  2. GET _remotestore/stats/<index_name>/<shard_id>

Path parameters

The following table lists the available path parameters. All path parameters are optional.

ParameterTypeDescription
index_nameStringThe index name or index pattern.
shard_idStringThe shard ID.

Remote store stats for an index

Use the following API to get remote store statistics for all shards of an index.

Example request

  1. GET _remotestore/stats/<index_name>

copy

Example response

Response

  1. {
  2.   "_shards": {
  3.     "total": 2,
  4.     "successful": 2,
  5.     "failed": 0
  6.   },
  7.   "stats": [
  8.     {
  9.       "shard_id": "[so][1]",
  10.       "refresh_time_lag_in_millis": 0,
  11.       "refresh_lag": 0,
  12.       "bytes_lag": 0,
  13.       "backpressure_rejection_count": 0,
  14.       "consecutive_failure_count": 0,
  15.       "total_remote_refresh": {
  16.         "started": 56,
  17.         "succeeded": 56,
  18.         "failed": 0
  19.       },
  20.       "total_uploads_in_bytes": {
  21.         "started": 1524670599,
  22.         "succeeded": 1524670599,
  23.         "failed": 0
  24.       },
  25.       "remote_refresh_size_in_bytes": {
  26.         "last_successful": 12711179,
  27.         "moving_avg": 30726409
  28.       },
  29.       "upload_latency_in_bytes_per_sec": {
  30.         "moving_avg": 25276841.3
  31.       },
  32.       "remote_refresh_latency_in_millis": {
  33.         "moving_avg": 964.7
  34.       }
  35.     },
  36.     {
  37.       "shard_id": "[so][0]",
  38.       "refresh_time_lag_in_millis": 5727,
  39.       "refresh_lag": 1,
  40.       "bytes_lag": 0,
  41.       "backpressure_rejection_count": 0,
  42.       "consecutive_failure_count": 0,
  43.       "total_remote_refresh": {
  44.         "started": 57,
  45.         "succeeded": 56,
  46.         "failed": 0
  47.       },
  48.       "total_uploads_in_bytes": {
  49.         "started": 1568138701,
  50.         "succeeded": 1568138701,
  51.         "failed": 0
  52.       },
  53.       "remote_refresh_size_in_bytes": {
  54.         "last_successful": 12705142,
  55.         "moving_avg": 32766119.75
  56.       },
  57.       "upload_latency_in_bytes_per_sec": {
  58.         "moving_avg": 25523682.95
  59.       },
  60.       "remote_refresh_latency_in_millis": {
  61.         "moving_avg": 990.55
  62.       }
  63.     }
  64.   ]
  65. }

Response fields

The following table lists the available response fields.

FieldDescription
refresh_time_lag_in_millisThe time (in milliseconds) the remote refresh is behind the local refresh.
refresh_lagThe number of refreshes by which the remote store is lagging behind the local store.
bytes_lagThe bytes lag between the remote and local store.
backpressure_rejection_countThe total number of write rejections made because of remote store backpressure.
consecutive_failure_countThe number of consecutive remote refresh failures since the last success.
total_remote_refreshThe total number of remote refreshes.
total_uploads_in_bytesThe total number of bytes in all uploads to the remote store.
remote_refresh_size_in_bytes.last_successfulThe size of data uploaded in the last successful refresh.
remote_refresh_size_in_bytes.moving_avgThe average size of data (in bytes) uploaded in the last N refreshes. N is defined in remote_store.segment.pressure.upload_bytes_moving_average_window_size. For details, see Remote segment backpressure.
upload_latency_in_bytes_per_sec.moving_avgThe average speed of remote store uploads (in bytes per second) for the last N uploads. N is defined in remote_store.segment.pressure.upload_bytes_per_sec_moving_average_window_size. For details, see Remote segment backpressure.
remote_refresh_latency_in_millis.moving_avgThe average time taken by a single remote refresh during the last N remote refreshes. N is defined in remote_store.segment.pressure.upload_time_moving_average_window_size. For details, see Remote segment backpressure.

Remote store stats for a single shard

Use the following API to get remote store statistics for a single shard.

Example request

  1. GET _remotestore/stats/<index_name>/<shard_id>

copy

Example response

Response

  1. {
  2.   "_shards": {
  3.     "total": 1,
  4.     "successful": 1,
  5.     "failed": 0
  6.   },
  7.   "stats": [
  8.     {
  9.       "shard_id": "[so][0]",
  10.       "refresh_time_lag_in_millis": 5727,
  11.       "refresh_lag": 1,
  12.       "bytes_lag": 0,
  13.       "backpressure_rejection_count": 0,
  14.       "consecutive_failure_count": 0,
  15.       "total_remote_refresh": {
  16.         "started": 57,
  17.         "succeeded": 56,
  18.         "failed": 0
  19.       },
  20.       "total_uploads_in_bytes": {
  21.         "started": 1568138701,
  22.         "succeeded": 1568138701,
  23.         "failed": 0
  24.       },
  25.       "remote_refresh_size_in_bytes": {
  26.         "last_successful": 12705142,
  27.         "moving_avg": 32766119.75
  28.       },
  29.       "upload_latency_in_bytes_per_sec": {
  30.         "moving_avg": 25523682.95
  31.       },
  32.       "remote_refresh_latency_in_millis": {
  33.         "moving_avg": 990.55
  34.       }
  35.     }
  36.   ]
  37. }

Remote store stats for a local shard

Provide the local query parameter set to true to only fetch the shards present on the node that is serving the request:

  1. GET _remotestore/stats/<index_name>?local=true

copy