Monitoring etcd

Each etcd server exports metrics under the /metrics path on its client port.

The metrics can be fetched with curl:

  1. $ curl -L http://localhost:2379/metrics
  2. # HELP etcd_debugging_mvcc_keys_total Total number of keys.
  3. # TYPE etcd_debugging_mvcc_keys_total gauge
  4. etcd_debugging_mvcc_keys_total 0
  5. # HELP etcd_debugging_mvcc_pending_events_total Total number of pending events to be sent.
  6. # TYPE etcd_debugging_mvcc_pending_events_total gauge
  7. etcd_debugging_mvcc_pending_events_total 0
  8. ...

Prometheus

Running a Prometheus monitoring service is the easiest way to ingest and record etcd’s metrics.

First, install Prometheus:

  1. PROMETHEUS_VERSION="1.3.1"
  2. wget https://github.com/prometheus/prometheus/releases/download/v$PROMETHEUS_VERSION/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz -O /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz
  3. tar -xvzf /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz --directory /tmp/ --strip-components=1
  4. /tmp/prometheus -version

Set Prometheus’s scraper to target the etcd cluster endpoints:

  1. cat > /tmp/test-etcd.yaml <<EOF
  2. global:
  3. scrape_interval: 10s
  4. scrape_configs:
  5. - job_name: test-etcd
  6. static_configs:
  7. - targets: ['10.240.0.32:2379','10.240.0.33:2379','10.240.0.34:2379']
  8. EOF
  9. cat /tmp/test-etcd.yaml

Set up the Prometheus handler:

  1. nohup /tmp/prometheus \
  2. -config.file /tmp/test-etcd.yaml \
  3. -web.listen-address ":9090" \
  4. -storage.local.path "test-etcd.data" >> /tmp/test-etcd.log 2>&1 &

Now Prometheus will scrape etcd metrics every 10 seconds.

Grafana

Grafana has built-in Prometheus support; just add a Prometheus data source:

  1. Name: test-etcd
  2. Type: Prometheus
  3. Url: http://localhost:9090
  4. Access: proxy

Then import the default etcd dashboard template and customize. For instance, if Prometheus data source name is my-etcd, the datasource field values in JSON also need to be my-etcd.

See the demo.

Sample dashboard:

监控(TBD) - 图1