09-3.部署 heapster 插件

Heapster是一个收集者，将每个Node上的cAdvisor的数据进行汇总，然后导到第三方工具(如InfluxDB)。

Heapster 是通过调用 kubelet 的 http API 来获取 cAdvisor 的 metrics 数据的。

由于 kubelet 只在 10250 端口接收 https 请求，故需要修改 heapster 的 deployment 配置。同时，需要赋予 kube-system:heapster ServiceAccount 调用 kubelet API 的权限。

注意：如果没有特殊指明，本文档的所有操作均在 m7-autocv-gpu01 节点上执行。

下载 heapster 文件

到 heapster release 页面 下载最新版本的 heapster

  1. cd /opt/k8s/work
  2. wget https://github.com/kubernetes/heapster/archive/v1.5.4.tar.gz
  3. tar -xzvf v1.5.4.tar.gz
  4. mv v1.5.4.tar.gz heapster-1.5.4.tar.gz

官方文件目录： heapster-1.5.4/deploy/kube-config/influxdb

修改配置

  1. $ cd heapster-1.5.4/deploy/kube-config/influxdb
  2. $ cp grafana.yaml{,.orig}
  3. $ diff grafana.yaml.orig grafana.yaml
  4. 67c67
  5. <   # type: NodePort
  6. ---
  7. >   type: NodePort
  • 开启 NodePort；
  1. $ cp heapster.yaml{,.orig}
  2. $ diff heapster.yaml.orig heapster.yaml
  3. <         - --source=kubernetes:https://kubernetes.default
  4. ---
  5. >         - --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250
  • 由于 kubelet 只在 10250 监听 https 请求，故添加相关参数；

执行所有定义文件

  1. $ cd  /opt/k8s/work/heapster-1.5.4/deploy/kube-config/influxdb
  2. $ ls *.yaml
  3. grafana.yaml  heapster.yaml  influxdb.yaml
  4. $ kubectl create -f  .
  6. $ cd ../rbac/
  7. $ cp heapster-rbac.yaml{,.orig}
  8. $ diff heapster-rbac.yaml.orig heapster-rbac.yaml
  9. 12a13,26
  10. > ---
  11. > kind: ClusterRoleBinding
  12. > apiVersion: rbac.authorization.k8s.io/v1beta1
  13. > metadata:
  14. >   name: heapster-kubelet-api
  15. > roleRef:
  16. >   apiGroup: rbac.authorization.k8s.io
  17. >   kind: ClusterRole
  18. >   name: system:kubelet-api-admin
  19. > subjects:
  20. > - kind: ServiceAccount
  21. >   name: heapster
  22. >   namespace: kube-system
  24. $ kubectl create -f heapster-rbac.yaml
  • 将 serviceAccount kube-system:heapster 与 ClusterRole system:kubelet-api-admin 绑定，授予它调用 kubelet API 的权限；

如果不修改，默认的 ClusterRole system:heapster 权限不足：

E1128 10:00:05.010716 1 manager.go:101] Error in scraping containers from kubelet:172.27.128.150:10250: failed to get all container stats from Kubelet URL “https://172.27.128.150:10250/stats/container/“: request failed - “403 Forbidden”, response: “Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)” E1128 10:00:05.018556 1 manager.go:101] Error in scraping containers from kubelet:172.27.128.149:10250: failed to get all container stats from Kubelet URL “https://172.27.128.149:10250/stats/container/“: request failed - “403 Forbidden”, response: “Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)” E1128 10:00:05.022664 1 manager.go:101] Error in scraping containers from kubelet:172.27.128.148:10250: failed to get all container stats from Kubelet URL “https://172.27.128.148:10250/stats/container/“: request failed - “403 Forbidden”, response: “Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)” W1128 10:00:25.000467 1 manager.go:152] Failed to get all responses in time (got 0/3)

检查执行结果

  1. $ kubectl get pods -n kube-system | grep -E 'heapster|monitoring'
  2. heapster-56c9dc749-j7hvz                1/1       Running   0          1m
  3. monitoring-grafana-c797777db-lnwnc      1/1       Running   0          1m
  4. monitoring-influxdb-cf9d95766-5wd28     1/1       Running   0          1m

检查 kubernets dashboard 界面，可以正确显示各 Nodes、Pods 的 CPU、内存、负载等统计数据和图表：

dashboard-heapster

访问 grafana

  1. 通过 kube-apiserver 访问：

    获取 monitoring-grafana 服务 URL：

    1.  $ kubectl cluster-info
    2.  Kubernetes master is running at https://127.0.0.1:8443
    3.  Heapster is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/heapster/proxy
    4.  CoreDNS is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
    5.  kubernetes-dashboard is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
    6.  monitoring-grafana is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
    7.  monitoring-influxdb is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
    9.  To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

    • 由于 apiserver 通过本地的 kube-nginx 做了代理，所以上面显示的 127.0.0.1:8443 为本地的 kube-nginx 的 IP 和 Port，浏览器访问时需要替换为 kube-apiserver 实际监听的 IP 和端口，如 172.27.128.149:6443；

      浏览器访问 URL： https://172.27.128.149:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy 对于 virtuabox 做了端口映射： http://127.0.0.1:8080/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

  2. 通过 kubectl proxy 访问：

    创建代理

    1.  kubectl proxy --address='172.27.128.150' --port=8086 --accept-hosts='^*$'
    2.  Starting to serve on 172.27.129.150:8086

    浏览器访问 URL：http://172.27.128.150:8086/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy/?orgId=1 对于 virtuabox 做了端口映射： http://127.0.0.1:8086/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy/?orgId=1

  3. 通过 NodePort 访问：

    1.  $ kubectl get svc -n kube-system|grep -E 'monitoring|heapster'
    2.  heapster               ClusterIP   10.254.199.65    <none>        80/TCP          3m
    3.  monitoring-grafana     NodePort    10.254.116.161   <none>        80:31470/TCP    3m
    4.  monitoring-influxdb    ClusterIP   10.254.250.185   <none>        8086/TCP        3m

    • grafana 监听 NodePort 31470；

      浏览器访问 URL：http://172.27.128.150:31470/?orgId=1

grafana

参考

  1. 配置 heapster：https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md