API

The M3 Coordinator implements the Prometheus Remote Read and Write HTTP endpoints, they also can be used however as general purpose metrics write and read APIs. Any metrics that are written to the remote write API can be queried using PromQL through the query APIs as well as being able to be read back by the Prometheus Remote Read endpoint.

Remote Write

Write a Prometheus Remote write query to M3.

URL

/api/v1/prom/remote/write

Method

POST

URL Params

None.

Header Params

Optional

  • M3-Metrics-Type:
    If this header is set, it determines what type of metric to store this metric value as. Otherwise by default, metrics will be stored in all namespaces that are configured. You can also disable this default behavior by setting downsample options to all: false for a namespace in the coordinator config, for more see disabling automatic aggregation. Must be one of:
    unaggregated: Write metrics directly to configured unaggregated namespace.
    aggregated: Write metrics directly to a configured aggregated namespace (bypassing any aggregation), this requires the M3-Storage-Policy header to be set to resolve which namespace to write metrics to.
  • M3-Storage-Policy:
    If this header is set, it determines which aggregated namespace to read/write metrics directly to/from (bypassing any aggregation).
    The value of the header must be in the format of resolution:retention in duration shorthand. e.g. 1m:48h specifices 1 minute resolution and 48 hour retention. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. Here is an example of querying metrics from a specific namespace.

  • M3-Map-Tags-JSON:
    If this header is set it enables dynamically mutating tags in a Prometheus write request. See issue 2254 for further context. Currently only write is supported. As an example, the following header would unconditionally cause globaltag=somevalue to be added to all metrics in a write request:

  1. M3-Map-Tags-JSON: '{"tagMappers":[{"write":{"tag":"globaltag","value":"somevalue"}}]}'

Data Params

Binary snappy compressed Prometheus WriteRequest protobuf message.

Available Tuning Params

Refer here for an up to date list of remote tuning parameters.

Sample Call

There isn’t a straightforward way to Snappy compress and marshal a Prometheus WriteRequest protobuf message using just shell, so this example uses a specific command line utility instead.

This sample call is made using promremotecli which is a command line tool that uses a Go client to Prometheus Remote endpoints. For more information visit the GitHub repository.

There is also a Java client that can be used to make requests to the endpoint.

Each -t parameter specifies a label (dimension) to add to the metric.

The -h parameter can be used as many times as necessary to add headers to the outgoing request in the form of “Header-Name: HeaderValue”.

Here is an example of writing the datapoint at the current unix timestamp with value 123.456:

  1. docker run -it --rm \
  2. quay.io/m3db/prometheus_remote_client_golang:latest \
  3. -u http://host.docker.internal:7201/api/v1/prom/remote/write \
  4. -t __name__:http_requests_total \
  5. -t code:200 \
  6. -t handler:graph \
  7. -t method:get \
  8. -d $(date +"%s"),123.456
  9. promremotecli_log 2019/06/25 04:13:56 writing datapoint [2019-06-25 04:13:55 +0000 UTC 123.456]
  10. promremotecli_log 2019/06/25 04:13:56 labelled [[__name__ http_requests_total] [code 200] [handler graph] [method get]]
  11. promremotecli_log 2019/06/25 04:13:56 writing to http://host.docker.internal:7201/api/v1/prom/remote/write
  12. {"success":true,"statusCode":200}
  13. promremotecli_log 2019/06/25 04:13:56 write success
  14. # If you are paranoid about image tags being hijacked/replaced with nefarious code, you can use this SHA256 tag:
  15. # quay.io/m3db/prometheus_remote_client_golang@sha256:fc56df819bff9a5a087484804acf3a584dd4a78c68900c31a28896ed66ca7e7b

For more details on querying data in PromQL that was written using this endpoint, see the query API documentation.

Remote Read

Read Prometheus metrics from M3.

URL

/api/v1/prom/remote/read

Method

POST

URL Params

None.

Header Params

Optional

  • M3-Metrics-Type:
    If this header is set, it determines what type of metric to store this metric value as. Otherwise by default, metrics will be stored in all namespaces that are configured. You can also disable this default behavior by setting downsample options to all: false for a namespace in the coordinator config, for more see disabling automatic aggregation. Must be one of:
    unaggregated: Write metrics directly to configured unaggregated namespace.
    aggregated: Write metrics directly to a configured aggregated namespace (bypassing any aggregation), this requires the M3-Storage-Policy header to be set to resolve which namespace to write metrics to.
  • M3-Storage-Policy:
    If this header is set, it determines which aggregated namespace to read/write metrics directly to/from (bypassing any aggregation).
    The value of the header must be in the format of resolution:retention in duration shorthand. e.g. 1m:48h specifices 1 minute resolution and 48 hour retention. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. Here is an example of querying metrics from a specific namespace.

  • M3-Restrict-By-Tags-JSON:
    If this header is set it can ensure specific label matching is performed as part of every query including series metadata endpoints. As an example, the following header would unconditionally cause globaltag=somevalue to be a part of all queries issued regardless of if they include the label or not in a query and also strip the “globaltag” from appearing as a label in any of the resulting timeseries:

  1. M3-Restrict-By-Tags-JSON: '{"match":[{"name":"globaltag","type":"EQUAL","value":"somevalue"}],"strip":["globaltag"]}'
  • M3-Limit-Max-Series:
    If this header is set it will override any configured per query time series limit. If the limit is hit, it will either return a partial result or an error based on the require exhaustive configuration set.

  • M3-Limit-Max-Docs:
    If this header is set it will override any configured per query time series * blocks limit (docs limit). If the limit is hit, it will either return a partial result or an error based on the require exhaustive configuration set.

  • M3-Limit-Require-Exhaustive:
    If this header is set it will override any configured require exhaustive setting. If “true” it will return an error if query hits a configured limit (such as series or docs limit) instead of a partial result. Otherwise if “false” it will return a partial result of the time series already matched with the response header M3-Results-Limited detailing the limit that was hit and a warning included in the response body.

Data Params

Binary snappy compressed Prometheus WriteRequest protobuf message.