Prometheus

This document is a getting started guide to integrating M3DB with Prometheus.

M3 Coordinator configuration

To write to a remote M3DB cluster the simplest configuration is to run m3coordinator as a sidecar alongside Prometheus.

Start by downloading the config template. Update the namespaces and the client section for a new cluster to match your cluster’s configuration.

You’ll need to specify the static IPs or hostnames of your M3DB seed nodes, and the name and retention values of the namespace you set up. You can leave the namespace storage metrics type as unaggregated since it’s required by default to have a cluster that receives all Prometheus metrics unaggregated. In the future you might also want to aggregate and downsample metrics for longer retention, and you can come back and update the config once you’ve setup those clusters. You can read more about our aggregation functionality here.

It should look something like:

  1. listenAddress: 0.0.0.0:7201
  3. logging:
  4.   level: info
  6. metrics:
  7.   scope:
  8.     prefix: "coordinator"
  9.   prometheus:
  10.     handlerPath: /metrics
  11.     listenAddress: 0.0.0.0:7203 # until https://github.com/m3db/m3/issues/682 is resolved
  12.   sanitization: prometheus
  13.   samplingRate: 1.0
  14.   extended: none
  16. tagOptions:
  17.   idScheme: quoted
  19. clusters:
  20.    - namespaces:
  21. # We created a namespace called "default" and had set it to retention "48h".
  22.        - namespace: default
  23.          retention: 48h
  24.          type: unaggregated
  25.      client:
  26.        config:
  27.          service:
  28.            env: default_env
  29.            zone: embedded
  30.            service: m3db
  31.            cacheDir: /var/lib/m3kv
  32.            etcdClusters:
  33.              - zone: embedded
  34.                endpoints:
  35. # We have five M3DB nodes but only three are seed nodes, they are listed here.
  36.                  - M3DB_NODE_01_STATIC_IP_ADDRESS:2379
  37.                  - M3DB_NODE_02_STATIC_IP_ADDRESS:2379
  38.                  - M3DB_NODE_03_STATIC_IP_ADDRESS:2379
  39.        writeConsistencyLevel: majority
  40.        readConsistencyLevel: unstrict_majority
  41.        writeTimeout: 10s
  42.        fetchTimeout: 15s
  43.        connectTimeout: 20s
  44.        writeRetry:
  45.          initialBackoff: 500ms
  46.          backoffFactor: 3
  47.          maxRetries: 2
  48.          jitter: true
  49.        fetchRetry:
  50.          initialBackoff: 500ms
  51.          backoffFactor: 2
  52.          maxRetries: 3
  53.          jitter: true
  54.        backgroundHealthCheckFailLimit: 4
  55.        backgroundHealthCheckFailThrottleFactor: 0.5

Now start the process up:

  1. m3coordinator -f <config-name.yml>

Or, use the docker container:

  1. docker pull quay.io/m3db/m3coordinator:latest
  2. docker run -p 7201:7201 --name m3coordinator -v <config-name.yml>:/etc/m3coordinator/m3coordinator.yml quay.io/m3db/m3coordinator:latest

Prometheus configuration

Add to your Prometheus configuration the m3coordinator sidecar remote read/write endpoints, something like:

  1. remote_read:
  2.   - url: "http://localhost:7201/api/v1/prom/remote/read"
  3.     # To test reading even when local Prometheus has the data
  4.     read_recent: true
  5. remote_write:
  6.   - url: "http://localhost:7201/api/v1/prom/remote/write"

Also, we recommend adding M3DB and M3Coordinator/M3Query to your list of jobs under scrape_configs so that you can monitor them using Prometheus. With this scraping setup, you can also use our pre-configured M3DB Grafana dashboard.

  1. - job_name: 'm3db'
  2.   static_configs:
  3.     - targets: ['<M3DB_HOST_NAME_1>:7203', '<M3DB_HOST_NAME_2>:7203', '<M3DB_HOST_NAME_3>:7203']
  4. - job_name: 'm3coordinator'
  5.   static_configs:
  6.     - targets: ['<M3COORDINATOR_HOST_NAME_1>:7203']

NOTE: If you are running M3DB with embedded M3Coordinator, you should only have one job. We recommend just calling this job m3. For example:

  1. - job_name: 'm3'
  2.   static_configs:
  3.     - targets: ['<HOST_NAME>:7203']

Querying With Grafana

When using the Prometheus integration with Grafana, there are two different ways you can query for your metrics. The first option is to configure Grafana to query Prometheus directly by following these instructions.

Alternatively, you can configure Grafana to read metrics directly from M3Coordinator in which case you will bypass Prometheus entirely and use M3’s PromQL engine instead. To set this up, follow the same instructions from the previous step, but set the url to: http://<M3_COORDINATOR_HOST_NAME>:7201.