使用etcdctl访问kubernetes数据

Kubenretes1.6中使用etcd V3版本的API,使用etcdctl直接ls的话只能看到/kube-centos一个路径。需要在命令前加上ETCDCTL_API=3这个环境变量才能看到kuberentes在etcd中保存的数据。

  1. ETCDCTL_API=3 etcdctl get /registry/namespaces/default -w=json|python -m json.tool

如果是使用 kubeadm 创建的集群,在 Kubenretes 1.11 中,etcd 默认使用 tls ,这时你可以在 master 节点上使用以下命令来访问 etcd :

  1. ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  2. --cert=/etc/kubernetes/pki/etcd/peer.crt \
  3. --key=/etc/kubernetes/pki/etcd/peer.key \
  4. get /registry/namespaces/default -w=json | jq .
  • -w指定输出格式

将得到这样的json的结果:

  1. {
  2. "count": 1,
  3. "header": {
  4. "cluster_id": 12091028579527406772,
  5. "member_id": 16557816780141026208,
  6. "raft_term": 36,
  7. "revision": 29253467
  8. },
  9. "kvs": [
  10. {
  11. "create_revision": 5,
  12. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==",
  13. "mod_revision": 5,
  14. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmIKSAoHZGVmYXVsdBIAGgAiACokZTU2YzMzMDgtMWVhOC0xMWU3LThjZDctZjRlOWQ0OWY4ZWQwMgA4AEILCIn4sscFEKOg9xd6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=",
  15. "version": 1
  16. }
  17. ]
  18. }

使用--prefix可以看到所有的子目录,如查看集群中的namespace:

  1. ETCDCTL_API=3 etcdctl get /registry/namespaces --prefix -w=json|python -m json.tool

输出结果中可以看到所有的namespace。

  1. {
  2. "count": 8,
  3. "header": {
  4. "cluster_id": 12091028579527406772,
  5. "member_id": 16557816780141026208,
  6. "raft_term": 36,
  7. "revision": 29253722
  8. },
  9. "kvs": [
  10. {
  11. "create_revision": 24310883,
  12. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYXV0b21vZGVs",
  13. "mod_revision": 24310883,
  14. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmQKSgoJYXV0b21vZGVsEgAaACIAKiQ1MjczOTU1ZC1iMzEyLTExZTctOTcwYy1mNGU5ZDQ5ZjhlZDAyADgAQgsI7fSWzwUQ6Jv1Z3oAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==",
  15. "version": 1
  16. },
  17. {
  18. "create_revision": 21387676,
  19. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYnJhbmQ=",
  20. "mod_revision": 21387676,
  21. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmEKRwoFYnJhbmQSABoAIgAqJGNkZmQ1Y2NmLWExYzktMTFlNy05NzBjLWY0ZTlkNDlmOGVkMDIAOABCDAjR9qLOBRDYn83XAXoAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==",
  22. "version": 1
  23. },
  24. {
  25. "create_revision": 5,
  26. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==",
  27. "mod_revision": 5,
  28. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmIKSAoHZGVmYXVsdBIAGgAiACokZTU2YzMzMDgtMWVhOC0xMWU3LThjZDctZjRlOWQ0OWY4ZWQwMgA4AEILCIn4sscFEKOg9xd6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=",
  29. "version": 1
  30. },
  31. {
  32. "create_revision": 18504694,
  33. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGV2",
  34. "mod_revision": 24310213,
  35. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmwKUgoDZGV2EgAaACIAKiQyOGRlMGVjNS04ZTEzLTExZTctOTcwYy1mNGU5ZDQ5ZjhlZDAyADgAQgwI89CezQUQ0v2fuQNaCwoEbmFtZRIDZGV2egASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA",
  36. "version": 4
  37. },
  38. {
  39. "create_revision": 10,
  40. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1wdWJsaWM=",
  41. "mod_revision": 10,
  42. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmcKTQoLa3ViZS1wdWJsaWMSABoAIgAqJGU1ZjhkY2I1LTFlYTgtMTFlNy04Y2Q3LWY0ZTlkNDlmOGVkMDIAOABCDAiJ+LLHBRDdrsDPA3oAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==",
  43. "version": 1
  44. },
  45. {
  46. "create_revision": 2,
  47. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1zeXN0ZW0=",
  48. "mod_revision": 2,
  49. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmYKTAoLa3ViZS1zeXN0ZW0SABoAIgAqJGU1NmFhMDVkLTFlYTgtMTFlNy04Y2Q3LWY0ZTlkNDlmOGVkMDIAOABCCwiJ+LLHBRDoq9ASegASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA",
  50. "version": 1
  51. },
  52. {
  53. "create_revision": 3774247,
  54. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvc3BhcmstY2x1c3Rlcg==",
  55. "mod_revision": 3774247,
  56. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEoABCmYKDXNwYXJrLWNsdXN0ZXISABoAIgAqJDMyNjY3ZDVjLTM0YWMtMTFlNy1iZmJkLThhZjFlM2E3YzViZDIAOABCDAiA1cbIBRDU3YuAAVoVCgRuYW1lEg1zcGFyay1jbHVzdGVyegASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA",
  57. "version": 1
  58. },
  59. {
  60. "create_revision": 15212191,
  61. "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMveWFybi1jbHVzdGVy",
  62. "mod_revision": 15212191,
  63. "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEn0KYwoMeWFybi1jbHVzdGVyEgAaACIAKiQ2YWNhNjk1Yi03N2Y5LTExZTctYmZiZC04YWYxZTNhN2M1YmQyADgAQgsI1qiKzAUQkoqxDloUCgRuYW1lEgx5YXJuLWNsdXN0ZXJ6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=",
  64. "version": 1
  65. }
  66. ]
  67. }

key的值是经过base64编码,需要解码后才能看到实际值,如:

  1. $ echo L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYXV0b21vZGVs|base64 -d
  2. /registry/namespaces/automodel

etcd中kubernetes的元数据

我们使用kubectl命令获取的kubernetes的对象状态实际上是保存在etcd中的,使用下面的脚本可以获取etcd中的所有kubernetes对象的key:

注意,我们使用了ETCD v3版本的客户端命令来访问etcd。

  1. #!/bin/bash
  2. # Get kubernetes keys from etcd
  3. export ETCDCTL_API=3
  4. keys=`etcdctl get /registry --prefix -w json|python -m json.tool|grep key|cut -d ":" -f2|tr -d '"'|tr -d ","`
  5. for x in $keys;do
  6. echo $x|base64 -d|sort
  7. done

通过输出的结果我们可以看到kubernetes的原数据是按何种结构包括在kuberentes中的,输出结果如下所示:

  1. /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-details-default
  2. /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-productpage-default
  3. /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-ratings-default
  4. ...
  5. /registry/configmaps/default/namerctl-script
  6. /registry/configmaps/default/namerd-config
  7. /registry/configmaps/default/nginx-config
  8. ...
  9. /registry/deployments/default/sdmk-page-sdmk
  10. /registry/deployments/default/sdmk-payment-web
  11. /registry/deployments/default/sdmk-report
  12. ...

我们可以看到所有的Kuberentes的所有元数据都保存在/registry目录下,下一层就是API对象类型(复数形式),再下一层是namespace,最后一层是对象的名字。

以下是etcd中存储的kubernetes所有的元数据类型:

  1. ThirdPartyResourceData
  2. apiextensions.k8s.io
  3. apiregistration.k8s.io
  4. certificatesigningrequests
  5. clusterrolebindings
  6. clusterroles
  7. configmaps
  8. controllerrevisions
  9. controllers
  10. daemonsets
  11. deployments
  12. events
  13. horizontalpodautoscalers
  14. ingress
  15. limitranges
  16. minions
  17. monitoring.coreos.com
  18. namespaces
  19. persistentvolumeclaims
  20. persistentvolumes
  21. poddisruptionbudgets
  22. pods
  23. ranges
  24. replicasets
  25. resourcequotas
  26. rolebindings
  27. roles
  28. secrets
  29. serviceaccounts
  30. services
  31. statefulsets
  32. storageclasses
  33. thirdpartyresources

参考