Prometheus

system-architecture

This document demonstrates how to scrape metrics from edge node through Yurt-Tunnel’s DNS mode within an OpenYurt cluster.

Environment

  • OpenYurt v0.5.0+

  • CoreDNS v1.6.8+

  • prometheus-operator

If you don’t have an OpenYurt on hand, you can use yurtctl to create one or convert from an exist Kubernetes cluster. Installation of prometheus-operator you can refer to kube-prometheus

1.Modify CoreDNS config

OpenYurt will create yurt-tunnel-nodes ConfigMap,which keeps track of nodename dns records of nodes.

1.mount yurt-tunnel-nodes to CoreDNS

  1. kubectl patch deployment coredns -n kube-system -p '{"spec": {"template": {"spec": {"volumes": [{"configMap":{"name":"yurt-tunnel-nodes"},"name": "edge"}]}}}}'
  2. kubectl patch deployment coredns -n kube-system -p '{"spec": { "template": { "spec": { "containers": [{"name":"coredns","volumeMounts": [{"mountPath": "/etc/edge", "name": "edge", "readOnly": true }]}]}}}}'

2.Modify CoreDNS config

use hosts plugin to load dns records in yurt-tunnel-nodes configmap.

  1. $ kubectl edit configmap coredns -n kube-system
  2. ...........
  3. Corefile: |
  4. .:53 {
  5. errors
  6. health {
  7. lameduck 5s
  8. }
  9. ready
  10. hosts /etc/edge/tunnel-nodes { # add hosts plugin
  11. reload 300ms
  12. fallthrough
  13. }
  14. kubernetes cluster.local in-addr.arpa ip6.arpa {
  15. pods insecure
  16. fallthrough in-addr.arpa ip6.arpa
  17. ttl 30
  18. }
  19. prometheus :9153
  20. forward . /etc/resolv.conf {
  21. max_concurrent 1000
  22. }
  23. cache 30
  24. loop
  25. reload
  26. loadbalance
  27. }

3.Restart CoreDNS

  1. kubectl patch deployment coredns -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","env":[{"name":"RESTART","value":"'$(date +%s)'"}]}]}}}}'

2.Config Prometheus

By default, prometheus scrape node metrics with node ip. With relabel functionality provided by prometheus, we can change node ip to node hostname. You can config scrape behavior by modify ServiceMonitor CR.

收集kubelet的metrics

Add relabel rule in kubelet ServiceMonitor,Use __meta_kubernetes_endpoint_address_target_name to replace node ip:

  1. $ kubectl edit serviceMonitor kubelet -n monitoring
  2. spec:
  3. endpoint:
  4. ..........
  5. relabelings:
  6. - action: replace # add relabel rule
  7. regex: (.*);.*:(.*)
  8. replacement: $1:$2
  9. sourceLabels:
  10. - __meta_kubernetes_endpoint_address_target_name
  11. - __address__
  12. targetLabel: __address__
  13. ..........

scape other metrics(take node-exporter as an example)

Yurt-tunnel will only do forward for port 10250 and 10255, if you want to add forward for other ports, you can modify yurt-tunnel-server-cfg ConfigMap. For node-exporter, you may need to add 9100 to https-proxy-ports。If you want to add http forward, just modify http-proxy-ports.

modify yurt-tunnel-server-cfgConfigMap

  1. kubectl patch configmap yurt-tunnel-server-cfg -n kube-system -p '{"data": {"https-proxy-ports":"9100"}}'

Add relabel rule in node-exporter ServiceMonitor,use __meta_kubernetes_pod_node_nameto replace node ip:

  1. $ kubectl edit servicemonitor prom-kube-prometheus-stack-node-exporter
  2. spec:
  3. endpoint:
  4. ......
  5. relabelings:
  6. - action: replace #add relabel rule
  7. regex: (.*);.*:(.*)
  8. replacement: $1:$2
  9. sourceLabels:
  10. - __meta_kubernetes_pod_node_name
  11. - __address__
  12. targetLabel: __address__
  13. ........

Reference

Openyurt Yurt-Tunnel DNS Mode