ConfigMap的热更新

ConfigMap是用来存储配置文件的kubernetes资源对象,所有的配置内容都存储在etcd中,下文主要是探究 ConfigMap 的创建和更新流程,以及对 ConfigMap 更新后容器内挂载的内容是否同步更新的测试。

测试示例

假设我们在 default namespace 下有一个名为 nginx-config 的 ConfigMap,可以使用 kubectl命令来获取:

  1. $ kubectl get configmap nginx-config
  2. NAME DATA AGE
  3. nginx-config 1 99d

获取该ConfigMap的内容。

  1. kubectl get configmap nginx-config -o yaml
  1. apiVersion: v1
  2. data:
  3. nginx.conf: |-
  4. worker_processes 1;
  5. events { worker_connections 1024; }
  6. http {
  7. sendfile on;
  8. server {
  9. listen 80;
  10. # a test endpoint that returns http 200s
  11. location / {
  12. proxy_pass http://httpstat.us/200;
  13. proxy_set_header X-Real-IP $remote_addr;
  14. }
  15. }
  16. server {
  17. listen 80;
  18. server_name api.hello.world;
  19. location / {
  20. proxy_pass http://l5d.default.svc.cluster.local;
  21. proxy_set_header Host $host;
  22. proxy_set_header Connection "";
  23. proxy_http_version 1.1;
  24. more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample';
  25. }
  26. }
  27. server {
  28. listen 80;
  29. server_name www.hello.world;
  30. location / {
  31. # allow 'employees' to perform dtab overrides
  32. if ($cookie_special_employee_cookie != "letmein") {
  33. more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample';
  34. }
  35. # add a dtab override to get people to our beta, world-v2
  36. set $xheader "";
  37. if ($cookie_special_employee_cookie ~* "dogfood") {
  38. set $xheader "/host/world => /srv/world-v2;";
  39. }
  40. proxy_set_header 'l5d-dtab' $xheader;
  41. proxy_pass http://l5d.default.svc.cluster.local;
  42. proxy_set_header Host $host;
  43. proxy_set_header Connection "";
  44. proxy_http_version 1.1;
  45. }
  46. }
  47. }
  48. kind: ConfigMap
  49. metadata:
  50. creationTimestamp: 2017-08-01T06:53:17Z
  51. name: nginx-config
  52. namespace: default
  53. resourceVersion: "14925806"
  54. selfLink: /api/v1/namespaces/default/configmaps/nginx-config
  55. uid: 18d70527-7686-11e7-bfbd-8af1e3a7c5bd

ConfigMap中的内容是存储到etcd中的,然后查询etcd:

  1. ETCDCTL_API=3 etcdctl get /registry/configmaps/default/nginx-config -w json|python -m json.tool

注意使用 v3 版本的 etcdctl API,下面是输出结果:

  1. {
  2. "count": 1,
  3. "header": {
  4. "cluster_id": 12091028579527406772,
  5. "member_id": 16557816780141026208,
  6. "raft_term": 36,
  7. "revision": 29258723
  8. },
  9. "kvs": [
  10. {
  11. "create_revision": 14925806,
  12. "key": "L3JlZ2lzdHJ5L2NvbmZpZ21hcHMvZGVmYXVsdC9uZ2lueC1jb25maWc=",
  13. "mod_revision": 14925806,
  14. "value": "azhzAAoPCgJ2MRIJQ29uZmlnTWFwEqQMClQKDG5naW54LWNvbmZpZxIAGgdkZWZhdWx0IgAqJDE4ZDcwNTI3LTc2ODYtMTFlNy1iZmJkLThhZjFlM2E3YzViZDIAOABCCwjdyoDMBRC5ss54egASywsKCm5naW54LmNvbmYSvAt3b3JrZXJfcHJvY2Vzc2VzIDE7CgpldmVudHMgeyB3b3JrZXJfY29ubmVjdGlvbnMgMTAyNDsgfQoKaHR0cCB7CiAgICBzZW5kZmlsZSBvbjsKCiAgICBzZXJ2ZXIgewogICAgICAgIGxpc3RlbiA4MDsKCiAgICAgICAgIyBhIHRlc3QgZW5kcG9pbnQgdGhhdCByZXR1cm5zIGh0dHAgMjAwcwogICAgICAgIGxvY2F0aW9uIC8gewogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9odHRwc3RhdC51cy8yMDA7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgIFgtUmVhbC1JUCAgJHJlbW90ZV9hZGRyOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgYXBpLmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKICAgICAgICAgICAgcHJveHlfcGFzcyBodHRwOi8vbDVkLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgSG9zdCAkaG9zdDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBDb25uZWN0aW9uICIiOwogICAgICAgICAgICBwcm94eV9odHRwX3ZlcnNpb24gMS4xOwoKICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgd3d3LmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKCgogICAgICAgICAgICAjIGFsbG93ICdlbXBsb3llZXMnIHRvIHBlcmZvcm0gZHRhYiBvdmVycmlkZXMKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgIT0gImxldG1laW4iKSB7CiAgICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgICAgICB9CgogICAgICAgICAgICAjIGFkZCBhIGR0YWIgb3ZlcnJpZGUgdG8gZ2V0IHBlb3BsZSB0byBvdXIgYmV0YSwgd29ybGQtdjIKICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIiOwoKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgfiogImRvZ2Zvb2QiKSB7CiAgICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIvaG9zdC93b3JsZCA9PiAvc3J2L3dvcmxkLXYyOyI7CiAgICAgICAgICAgIH0KCiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgJ2w1ZC1kdGFiJyAkeGhlYWRlcjsKCgogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9sNWQuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBIb3N0ICRob3N0OwogICAgICAgICAgICBwcm94eV9zZXRfaGVhZGVyIENvbm5lY3Rpb24gIiI7CiAgICAgICAgICAgIHByb3h5X2h0dHBfdmVyc2lvbiAxLjE7CiAgICAgICAgfQogICAgfQp9GgAiAA==",
  15. "version": 1
  16. }
  17. ]
  18. }

其中的value就是 nginx.conf 配置文件的内容。

可以使用base64解码查看具体值,关于etcdctl的使用请参考使用etcdctl访问kuberentes数据

代码

ConfigMap 结构体的定义:

  1. // ConfigMap holds configuration data for pods to consume.
  2. type ConfigMap struct {
  3. metav1.TypeMeta `json:",inline"`
  4. // Standard object's metadata.
  5. // More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
  6. // +optional
  7. metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
  8. // Data contains the configuration data.
  9. // Each key must be a valid DNS_SUBDOMAIN with an optional leading dot.
  10. // +optional
  11. Data map[string]string `json:"data,omitempty" protobuf:"bytes,2,rep,name=data"`
  12. }

staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go 中ConfigMap 的接口定义:

  1. // ConfigMapInterface has methods to work with ConfigMap resources.
  2. type ConfigMapInterface interface {
  3. Create(*v1.ConfigMap) (*v1.ConfigMap, error)
  4. Update(*v1.ConfigMap) (*v1.ConfigMap, error)
  5. Delete(name string, options *meta_v1.DeleteOptions) error
  6. DeleteCollection(options *meta_v1.DeleteOptions, listOptions meta_v1.ListOptions) error
  7. Get(name string, options meta_v1.GetOptions) (*v1.ConfigMap, error)
  8. List(opts meta_v1.ListOptions) (*v1.ConfigMapList, error)
  9. Watch(opts meta_v1.ListOptions) (watch.Interface, error)
  10. Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.ConfigMap, err error)
  11. ConfigMapExpansion
  12. }

staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go 中创建 ConfigMap 的方法如下:

  1. // Create takes the representation of a configMap and creates it. Returns the server's representation of the configMap, and an error, if there is any.
  2. func (c *configMaps) Create(configMap *v1.ConfigMap) (result *v1.ConfigMap, err error) {
  3. result = &v1.ConfigMap{}
  4. err = c.client.Post().
  5. Namespace(c.ns).
  6. Resource("configmaps").
  7. Body(configMap).
  8. Do().
  9. Into(result)
  10. return
  11. }

通过 RESTful 请求在 etcd 中存储 ConfigMap 的配置,该方法中设置了资源对象的 namespace 和 HTTP 请求中的 body,执行后将请求结果保存到 result 中返回给调用者。

注意 Body 的结构

  1. // Body makes the request use obj as the body. Optional.
  2. // If obj is a string, try to read a file of that name.
  3. // If obj is a []byte, send it directly.
  4. // If obj is an io.Reader, use it directly.
  5. // If obj is a runtime.Object, marshal it correctly, and set Content-Type header.
  6. // If obj is a runtime.Object and nil, do nothing.
  7. // Otherwise, set an error.

创建 ConfigMap RESTful 请求中的的 Body 中包含 ObjectMetanamespace

HTTP 请求中的结构体:

  1. // Request allows for building up a request to a server in a chained fashion.
  2. // Any errors are stored until the end of your call, so you only have to
  3. // check once.
  4. type Request struct {
  5. // required
  6. client HTTPClient
  7. verb string
  8. baseURL *url.URL
  9. content ContentConfig
  10. serializers Serializers
  11. // generic components accessible via method setters
  12. pathPrefix string
  13. subpath string
  14. params url.Values
  15. headers http.Header
  16. // structural elements of the request that are part of the Kubernetes API conventions
  17. namespace string
  18. namespaceSet bool
  19. resource string
  20. resourceName string
  21. subresource string
  22. timeout time.Duration
  23. // output
  24. err error
  25. body io.Reader
  26. // This is only used for per-request timeouts, deadlines, and cancellations.
  27. ctx context.Context
  28. backoffMgr BackoffManager
  29. throttle flowcontrol.RateLimiter
  30. }

测试

分别测试使用 ConfigMap 挂载 Env 和 Volume 的情况。

更新使用ConfigMap挂载的Env

使用下面的配置创建 nginx 容器测试更新 ConfigMap 后容器内的环境变量是否也跟着更新。

  1. apiVersion: extensions/v1beta1
  2. kind: Deployment
  3. metadata:
  4. name: my-nginx
  5. spec:
  6. replicas: 1
  7. template:
  8. metadata:
  9. labels:
  10. run: my-nginx
  11. spec:
  12. containers:
  13. - name: my-nginx
  14. image: harbor-001.jimmysong.io/library/nginx:1.9
  15. ports:
  16. - containerPort: 80
  17. envFrom:
  18. - configMapRef:
  19. name: env-config
  20. ---
  21. apiVersion: v1
  22. kind: ConfigMap
  23. metadata:
  24. name: env-config
  25. namespace: default
  26. data:
  27. log_level: INFO

获取环境变量的值

  1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level
  2. log_level=INFO

修改 ConfigMap

  1. $ kubectl edit configmap env-config

修改 log_level 的值为 DEBUG

再次查看环境变量的值。

  1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level
  2. log_level=INFO

实践证明修改 ConfigMap 无法更新容器中已注入的环境变量信息。

更新使用ConfigMap挂载的Volume

使用下面的配置创建 nginx 容器测试更新 ConfigMap 后容器内挂载的文件是否也跟着更新。

  1. apiVersion: extensions/v1beta1
  2. kind: Deployment
  3. metadata:
  4. name: my-nginx
  5. spec:
  6. replicas: 1
  7. template:
  8. metadata:
  9. labels:
  10. run: my-nginx
  11. spec:
  12. containers:
  13. - name: my-nginx
  14. image: harbor-001.jimmysong.io/library/nginx:1.9
  15. ports:
  16. - containerPort: 80
  17. volumeMounts:
  18. - name: config-volume
  19. mountPath: /etc/config
  20. volumes:
  21. - name: config-volume
  22. configMap:
  23. name: special-config
  24. ---
  25. apiVersion: v1
  26. kind: ConfigMap
  27. metadata:
  28. name: special-config
  29. namespace: default
  30. data:
  31. log_level: INFO
  1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /etc/config/log_level
  2. INFO

修改 ConfigMap

  1. $ kubectl edit configmap special-config

修改 log_level 的值为 DEBUG

等待大概10秒钟时间,再次查看环境变量的值。

  1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /tmp/log_level
  2. DEBUG

我们可以看到使用 ConfigMap 方式挂载的 Volume 的文件中的内容已经变成了 DEBUG

Known Issue:如果使用ConfigMap的subPath挂载为Container的Volume,Kubernetes不会做自动热更新:https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

ConfigMap 更新后滚动更新 Pod

更新 ConfigMap 目前并不会触发相关 Pod 的滚动更新,可以通过修改 pod annotations 的方式强制触发滚动更新。

  1. $ kubectl patch deployment my-nginx --patch '{"spec": {"template": {"metadata": {"annotations": {"version/config": "20180411" }}}}}'

这个例子里我们在 .spec.template.metadata.annotations 中添加 version/config,每次通过修改 version/config 来触发滚动更新。

总结

更新 ConfigMap 后:

  • 使用该 ConfigMap 挂载的 Env 不会同步更新
  • 使用该 ConfigMap 挂载的 Volume 中的数据需要一段时间(实测大概10秒)才能同步更新

ENV 是在容器启动的时候注入的,启动之后 kubernetes 就不会再改变环境变量的值,且同一个 namespace 中的 pod 的环境变量是不断累加的,参考 Kubernetes中的服务发现与docker容器间的环境变量传递源码探究。为了更新容器中使用 ConfigMap 挂载的配置,需要通过滚动更新 pod 的方式来强制重新挂载 ConfigMap。

参考