Static File Service Discovery

How Prometheus discovery monitoring targets with static files

When prometheus_sd_method == ‘static’, Prometheus will use static conf files to manage monitoring targets.

Prometheus FHS

Monitoring Targets are placed @ /etc/prometheus/targets/, And divided by modules into several sub directories: infra, nodes, pgsql, redis, ….

  1. #------------------------------------------------------------------------------
  2. # Config FHS
  3. #------------------------------------------------------------------------------
  4. # /etc/prometheus/
  5. #  ^-----prometheus.yml              # prometheus main config file
  6. #  ^-----alertmanager.yml            # alertmanger main config file
  7. #  ^-----@bin                        # util scripts: check,reload,status,new
  8. #  ^-----@rules                      # record & alerting rules definition
  9. #            ^-----@infra            # infrastructure rules & alert
  10. #            ^-----@nodes            # nodes rules & alert
  11. #            ^-----@pgsql            # pgsql rules & alert
  12. #            ^-----@redis            # redis rules & alert
  13. #            ^-----@..........       # etc...
  14. #  ^-----@targets                    # file based service discovery targets definition
  15. #            ^-----@infra            # infra static targets definition
  16. #            ^-----@nodes            # nodes static targets definition
  17. #            ^-----@pgsql            # pgsql static targets definition
  18. #            ^-----@redis            # redis static targets definition
  19. #            ^-----@.....            # other targets
  20. #------------------------------------------------------------------------------

Targets

  1. /etc/prometheus/targets
  2. ├── nodes
  3. │   ├── 10.10.10.13.yml
  4. │   ├── 10.10.10.10.yml
  5. │   ├── 10.10.10.11.yml
  6. │   ├── 10.10.10.12.yml
  7. ├── pgsql
  8. │   ├── pg-meta-1.yml
  9. │   ├── pg-test-2.yml
  10. │   ├── pg-test-3.yml
  11. │   ├── pg-test-1.yml
  12. ├── infra
  13. │   ├── 10.10.10.10.yml
  14. ├── redis
  15. │   ├── redis-common-1.yml
  16. │   ├── redis-test-2.yml
  17. │   ├── redis-meta-1.yml
  18. │   ├── redis-test-1.yml

INFRA Targets

INFRA监控对象以元节点的IP地址作为文件名，内容如下：

  1. ---
  2. #------------------------------------------------------------------------------
  3. # Prometheus (9090)
  4. #------------------------------------------------------------------------------
  5. - labels: { ip: 10.10.10.10, type: prometheus }
  6.   targets:
  7.     - 10.10.10.10:9090
  9. #------------------------------------------------------------------------------
  10. # AlertManager (9093)
  11. #------------------------------------------------------------------------------
  12. - labels: { ip: 10.10.10.10, type: alertmanager }
  13.   targets:
  14.     - 10.10.10.10:9093
  16. #------------------------------------------------------------------------------
  17. # Grafana (3000)
  18. #------------------------------------------------------------------------------
  19. - labels: { ip: 10.10.10.10, type: grafana }
  20.   targets:
  21.     - 10.10.10.10:3000
  23. #------------------------------------------------------------------------------
  24. # Loki (3100)
  25. #------------------------------------------------------------------------------
  26. - labels: { ip: 10.10.10.10, type: loki }
  27.   targets:
  28.     - 10.10.10.10:3100
  30. #------------------------------------------------------------------------------
  31. # Nginx (Exporter @ 9113)
  32. #------------------------------------------------------------------------------
  33. - labels: { ip: 10.10.10.10, type: nginx }
  34.   targets:
  35.     - 10.10.10.10:9113

DCS Targets

DCS监控对象直接定于 /etc/prometheus/prometheus.yml 主配置文件中：

  1.   - job_name: consul
  2.     metrics_path: /v1/agent/metrics
  3.     params:
  4.       format: ['prometheus']
  5.     static_configs: [ { targets: [ 127.0.0.1:8500 ] , labels: { ip: 10.10.10.10, type: consul , job: infra } } ]
  7.   - job_name: etcd
  8.     metrics_path: /metrics
  9.     static_configs:
  10.       - { targets: [ 10.10.10.11:2379 ] , labels: { ip: 10.10.10.11, ins: meta-2, type: etcd , job: infra } }
  11.       - { targets: [ 10.10.10.12:2379 ] , labels: { ip: 10.10.10.12, ins: meta-3, type: etcd , job: infra } }
  12.       - { targets: [ 10.10.10.10:2379 ] , labels: { ip: 10.10.10.10, ins: meta-1, type: etcd , job: infra } }

NODES Targets

NODES监控对象以IP地址作为监控对象文件名，内容如下所示：

  1. $ cat nodes/10.10.10.10.yml
  1. # 10.10.10.10
  2. - labels: { ip: 10.10.10.10 , ins: pg-meta-1 , cls: pg-meta }
  3.   targets:
  4.     - 10.10.10.10:9100   # node_exporter
  5.     - 10.10.10.10:9323   # docker
  6.     - 10.10.10.10:9080   # promtail

PGSQL Targets

PGSQL监控对象以实例名作为监控对象文件名，内容如下所示：

  1. $ cat pgsql/pg-meta-1.yml
  1. # pg-meta-1 [primary] @ 10.10.10.10
  2. - labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
  3.   targets:
  4.     - 10.10.10.10:9630  # postgres
  5.     - 10.10.10.10:9631  # pgbouncer
  6.     - 10.10.10.10:8008  # patroni
  7.     - 10.10.10.10:9101  # haproxy

REDIS Targets

REDIS监控对象以Redis节点名作为监控对象文件名，内容如下所示：

  1. $ cat pgsql/redis-common-1.yml
  1. # redis-common-1 @ 10.10.10.13
  3. - labels: { cls: redis-common, ins: redis-common-1-6501, instance: 10.10.10.13:6501 }
  4.   targets: [ redis://10.10.10.13:6501 ]
  6. - labels: { cls: redis-common, ins: redis-common-1-6502, instance: 10.10.10.13:6502 }
  7.   targets: [ redis://10.10.10.13:6502 ]
  9. - labels: { cls: redis-common, ins: redis-common-1-6503, instance: 10.10.10.13:6503 }
  10.   targets: [ redis://10.10.10.13:6503 ]

Last modified 2022-06-04: fill en docs (5a858d3)