本文讲解了如何使您的集群符合互联网安全中心发布的 Kubernetes 安全基准,保护集群中节点的安全。安装 Kubernetes 之前,请按照本指南进行操作。加固指南旨在与特定版本的 CIS Kubernetes Benchmark,Kubernetes 和 Rancher 一起使用。

加固指南旨在与特定版本的 CIS Kubernetes Benchmark,Kubernetes 和 Rancher 一起使用:

加固指南版本Rancher 版本CIS Benchmark 版本Kubernetes 版本
加固指南 v2.3.5Rancher v2.3.5Benchmark v1.5Kubernetes v1.15

点击这里下载 PDF 版本的加固指南

概览

下面的安全加固指南是针对在生产环境的 Rancher v2.3.5 中使用 Kubernetes v1.15 版本的集群。它概述了如何满足互联网安全中心(CIS)提出的 Kubernetes 安全标准。

有关如果根据官方 CIS 基准评估集群的更多详细信息,请参阅CIS Benchmark Rancher 自测指南 - Rancher v2.3.5

已知问题

如果注册自定义节点时只提供了公共 IP,在 CIS 1.5 加固设置中,将无法正常在 Rancher UI 中使用执行命令行查看日志功能。如果想要使用上述两个功能,请在注册自定义节点时提供私有 IP 地址。

  • 如果注册自定义节点时只提供了公共 IP,在 CIS 1.5 加固设置中,将无法正常在 Rancher UI 中使用执行命令行查看日志功能。如果想要使用上述两个功能,请在注册自定义节点时提供私有 IP 地址。
  • default_pod_security_policy_template_id:restricted时,Rancher 在默认的 service account 中创建角色绑定集群角色绑定。CIS 1.5 要求默认 service account 没有绑定任何角色,不提供 service account 的 token,不分配特定的权限。

配置内核运行时参数

对于集群中的所有类型的节点,都建议使用以下的sysctl配置。在/etc/sysctl.d/90-kubelet.conf中设置以下参数:

  1. vm.overcommit_memory=1
  2. vm.panic_on_oom=0
  3. kernel.panic=10
  4. kernel.panic_on_oops=1
  5. kernel.keys.root_maxbytes=25000000

执行sysctl -p /etc/sysctl.d/90-kubelet.conf来启用配置。

配置etcd用户和组

在安装 RKE 之前,需要设置etcd服务的用户帐户和组。etcd用户的uidgid将在 RKE 的config.yml中使用,以在安装期间为文件和目录设置适当的权限。

创建etcd用户和组

要创建etcd组,请运行以下控制台命令。

  1. addgroup --gid 52034 etcd
  2. useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd

使用etcd用户的uidgid更新 RKE 的 config.yml文件:

  1. services:
  2. etcd:
  3. gid: 52034
  4. uid: 52034

default服务账号的automountServiceAccountToken设置为 false。

Kubernetes 提供了一个默认服务账号(Service Account),如果集群的工作负载中没有为 Pod 分配任何特定服务账号,那么它将会使用这个default的服务账号。在需要从 Pod 访问 Kubernetes API 的情况下,应为该 Pod 创建一个特定的服务账号,并向该服务账号授予权限。这个default的服务账户应该被设置为不提供服务账号令牌(service account token)和任何权限。将automountServiceAccountToken设置为 false 之后,Kubernetes 在启动 Pod 时,将不会自动注入default服务账户。

对于每个命名空间中的,default服务账号必须包含以下值:

  1. automountServiceAccountToken: false

把下面的 yaml 另存为account_update.yaml

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4. name: default
  5. automountServiceAccountToken: false

创建一个名称为account_update.sh的脚本。通过运行chmod +x account_update.sh,使这个脚本有执行权限。

  1. #!/bin/bash -e
  2. for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
  3. kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
  4. done

确保所有命名空间均已定义网络策略

在同一个 Kubernetes 集群上运行不同的应用程序会产生一个风险,那就是应用可能受到相邻应用程序的攻击。为了确保容器只能与预期的容器进行通信,网络细分是必不可少的。通过设置网络策略(Network Policy),可以设置哪些 Pod 之间可以通信,以及是否可以和其他网络端点进行通信。

网络策略是作用于命名空间范围的。将网络策略应用于给定命名空间时,所有不被这个策略允许的流量将被拒绝。然而,如果命名空间中没有设置网络策略,那么进出这个命名空间中 Pod 的所有流量都将被允许。要使用网络策略,必须启用 CNI(容器网络接口)插件。本指南使用canal提供策略实施。您可以在这里找到有关 CNI 插件的其他信息。

在集群上启用 CN​​I 插件后,您可以设置一个默认的网络策略。下面是一个宽松的网络策略示例,仅供参考。如果您想要允许到某个命名空间内所有 Pod 的流量(即使已经添加了一些策略,使得一些 Pods 被隔离了),您可以创建一个策略,明确允许该命名空间中的所有流量。将以下yaml另存为 default-allow-all.yaml。额外关于网络策略的信息,请查看Kubernetes 官方文档

:::important 重要 这个NetworkPolicy示例不建议在生产环境中使用。 :::

  1. ---
  2. apiVersion: networking.k8s.io/v1
  3. kind: NetworkPolicy
  4. metadata:
  5. name: default-allow-all
  6. spec:
  7. podSelector: {}
  8. ingress:
  9. - {}
  10. egress:
  11. - {}
  12. policyTypes:
  13. - Ingress
  14. - Egress

创建一个名称为apply_networkPolicy_to_all_ns.sh的脚本。通过运行chmod +x apply_networkPolicy_to_all_ns.sh,使这个脚本有执行权限。

  1. #!/bin/bash -e
  2. for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
  3. kubectl apply -f default-allow-all.yaml -n ${namespace}
  4. done

运行脚本,以使全部的命名空间使用这个default-allow-all.yaml文件中的宽松NetworkPolicy

加固的 RKE cluster.yml 配置参考

您可以用这个供您参考的cluster.yml,通过 RKE CLI 来创建安全加固的 Rancher Kubernetes Engine(RKE)集群。有关每个配置的详细信息,请参阅RKE 文档

  1. # 如果您打算在离线环境中部署Kubernetes,
  2. # 请查阅有关如何配置自定义RKE镜像的文档。
  3. kubernetes_version: "v1.15.9-rancher1-1"
  4. enable_network_policy: true
  5. default_pod_security_policy_template_id: "restricted"
  6. services:
  7. etcd:
  8. uid: 52034
  9. gid: 52034
  10. kube-api:
  11. pod_security_policy: true
  12. secrets_encryption_config:
  13. enabled: true
  14. audit_log:
  15. enabled: true
  16. admission_configuration:
  17. event_rate_limit:
  18. enabled: true
  19. kube-controller:
  20. extra_args:
  21. feature-gates: "RotateKubeletServerCertificate=true"
  22. scheduler:
  23. image: ""
  24. extra_args: {}
  25. extra_binds: []
  26. extra_env: []
  27. kubelet:
  28. generate_serving_certificate: true
  29. extra_args:
  30. feature-gates: "RotateKubeletServerCertificate=true"
  31. protect-kernel-defaults: "true"
  32. tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
  33. extra_binds: []
  34. extra_env: []
  35. cluster_domain: ""
  36. infra_container_image: ""
  37. cluster_dns_server: ""
  38. fail_swap_on: false
  39. kubeproxy:
  40. image: ""
  41. extra_args: {}
  42. extra_binds: []
  43. extra_env: []
  44. network:
  45. plugin: ""
  46. options: {}
  47. mtu: 0
  48. node_selector: {}
  49. authentication:
  50. strategy: ""
  51. sans: []
  52. webhook: null
  53. addons: |
  54. ---
  55. apiVersion: v1
  56. kind: Namespace
  57. metadata:
  58. name: ingress-nginx
  59. ---
  60. apiVersion: rbac.authorization.k8s.io/v1
  61. kind: Role
  62. metadata:
  63. name: default-psp-role
  64. namespace: ingress-nginx
  65. rules:
  66. - apiGroups:
  67. - extensions
  68. resourceNames:
  69. - default-psp
  70. resources:
  71. - podsecuritypolicies
  72. verbs:
  73. - use
  74. ---
  75. apiVersion: rbac.authorization.k8s.io/v1
  76. kind: RoleBinding
  77. metadata:
  78. name: default-psp-rolebinding
  79. namespace: ingress-nginx
  80. roleRef:
  81. apiGroup: rbac.authorization.k8s.io
  82. kind: Role
  83. name: default-psp-role
  84. subjects:
  85. - apiGroup: rbac.authorization.k8s.io
  86. kind: Group
  87. name: system:serviceaccounts
  88. - apiGroup: rbac.authorization.k8s.io
  89. kind: Group
  90. name: system:authenticated
  91. ---
  92. apiVersion: v1
  93. kind: Namespace
  94. metadata:
  95. name: cattle-system
  96. ---
  97. apiVersion: rbac.authorization.k8s.io/v1
  98. kind: Role
  99. metadata:
  100. name: default-psp-role
  101. namespace: cattle-system
  102. rules:
  103. - apiGroups:
  104. - extensions
  105. resourceNames:
  106. - default-psp
  107. resources:
  108. - podsecuritypolicies
  109. verbs:
  110. - use
  111. ---
  112. apiVersion: rbac.authorization.k8s.io/v1
  113. kind: RoleBinding
  114. metadata:
  115. name: default-psp-rolebinding
  116. namespace: cattle-system
  117. roleRef:
  118. apiGroup: rbac.authorization.k8s.io
  119. kind: Role
  120. name: default-psp-role
  121. subjects:
  122. - apiGroup: rbac.authorization.k8s.io
  123. kind: Group
  124. name: system:serviceaccounts
  125. - apiGroup: rbac.authorization.k8s.io
  126. kind: Group
  127. name: system:authenticated
  128. ---
  129. apiVersion: policy/v1beta1
  130. kind: PodSecurityPolicy
  131. metadata:
  132. name: restricted
  133. spec:
  134. requiredDropCapabilities:
  135. - NET_RAW
  136. privileged: false
  137. allowPrivilegeEscalation: false
  138. defaultAllowPrivilegeEscalation: false
  139. fsGroup:
  140. rule: RunAsAny
  141. runAsUser:
  142. rule: MustRunAsNonRoot
  143. seLinux:
  144. rule: RunAsAny
  145. supplementalGroups:
  146. rule: RunAsAny
  147. volumes:
  148. - emptyDir
  149. - secret
  150. - persistentVolumeClaim
  151. - downwardAPI
  152. - configMap
  153. - projected
  154. ---
  155. apiVersion: rbac.authorization.k8s.io/v1
  156. kind: ClusterRole
  157. metadata:
  158. name: psp:restricted
  159. rules:
  160. - apiGroups:
  161. - extensions
  162. resourceNames:
  163. - restricted
  164. resources:
  165. - podsecuritypolicies
  166. verbs:
  167. - use
  168. ---
  169. apiVersion: rbac.authorization.k8s.io/v1
  170. kind: ClusterRoleBinding
  171. metadata:
  172. name: psp:restricted
  173. roleRef:
  174. apiGroup: rbac.authorization.k8s.io
  175. kind: ClusterRole
  176. name: psp:restricted
  177. subjects:
  178. - apiGroup: rbac.authorization.k8s.io
  179. kind: Group
  180. name: system:serviceaccounts
  181. - apiGroup: rbac.authorization.k8s.io
  182. kind: Group
  183. name: system:authenticated
  184. ---
  185. apiVersion: v1
  186. kind: ServiceAccount
  187. metadata:
  188. name: tiller
  189. namespace: kube-system
  190. ---
  191. apiVersion: rbac.authorization.k8s.io/v1
  192. kind: ClusterRoleBinding
  193. metadata:
  194. name: tiller
  195. roleRef:
  196. apiGroup: rbac.authorization.k8s.io
  197. kind: ClusterRole
  198. name: cluster-admin
  199. subjects:
  200. - kind: ServiceAccount
  201. name: tiller
  202. namespace: kube-system
  203. addons_include: []
  204. system_images:
  205. etcd: ""
  206. alpine: ""
  207. nginx_proxy: ""
  208. cert_downloader: ""
  209. kubernetes_services_sidecar: ""
  210. kubedns: ""
  211. dnsmasq: ""
  212. kubedns_sidecar: ""
  213. kubedns_autoscaler: ""
  214. coredns: ""
  215. coredns_autoscaler: ""
  216. kubernetes: ""
  217. flannel: ""
  218. flannel_cni: ""
  219. calico_node: ""
  220. calico_cni: ""
  221. calico_controllers: ""
  222. calico_ctl: ""
  223. calico_flexvol: ""
  224. canal_node: ""
  225. canal_cni: ""
  226. canal_flannel: ""
  227. canal_flexvol: ""
  228. weave_node: ""
  229. weave_cni: ""
  230. pod_infra_container: ""
  231. ingress: ""
  232. ingress_backend: ""
  233. metrics_server: ""
  234. windows_pod_infra_container: ""
  235. ssh_key_path: ""
  236. ssh_cert_path: ""
  237. ssh_agent_auth: false
  238. authorization:
  239. mode: ""
  240. options: {}
  241. ignore_docker_version: false
  242. private_registries: []
  243. ingress:
  244. provider: ""
  245. options: {}
  246. node_selector: {}
  247. extra_args: {}
  248. dns_policy: ""
  249. extra_envs: []
  250. extra_volumes: []
  251. extra_volume_mounts: []
  252. cluster_name: ""
  253. prefix_path: ""
  254. addon_job_timeout: 0
  255. bastion_host:
  256. address: ""
  257. port: ""
  258. user: ""
  259. ssh_key: ""
  260. ssh_key_path: ""
  261. ssh_cert: ""
  262. ssh_cert_path: ""
  263. monitoring:
  264. provider: ""
  265. options: {}
  266. node_selector: {}
  267. restore:
  268. restore: false
  269. snapshot_name: ""
  270. dns: null

安全加固的 RKE 模板配置参考

这个 RKE 参考模板提供了安装安全加固的 Kubenetes 所需的配置。RKE 模板用于配置 Kubernetes 和定义 Rancher 设置。请参阅Rancher 文档获得更多安装和 RKE 模板的详细信息。

  1. #
  2. # Cluster Config
  3. #
  4. default_pod_security_policy_template_id: restricted
  5. docker_root_dir: /var/lib/docker
  6. enable_cluster_alerting: false
  7. enable_cluster_monitoring: false
  8. enable_network_policy: true
  9. #
  10. # Rancher Config
  11. #
  12. rancher_kubernetes_engine_config:
  13. addon_job_timeout: 30
  14. addons: |-
  15. ---
  16. apiVersion: v1
  17. kind: Namespace
  18. metadata:
  19. name: ingress-nginx
  20. ---
  21. apiVersion: rbac.authorization.k8s.io/v1
  22. kind: Role
  23. metadata:
  24. name: default-psp-role
  25. namespace: ingress-nginx
  26. rules:
  27. - apiGroups:
  28. - extensions
  29. resourceNames:
  30. - default-psp
  31. resources:
  32. - podsecuritypolicies
  33. verbs:
  34. - use
  35. ---
  36. apiVersion: rbac.authorization.k8s.io/v1
  37. kind: RoleBinding
  38. metadata:
  39. name: default-psp-rolebinding
  40. namespace: ingress-nginx
  41. roleRef:
  42. apiGroup: rbac.authorization.k8s.io
  43. kind: Role
  44. name: default-psp-role
  45. subjects:
  46. - apiGroup: rbac.authorization.k8s.io
  47. kind: Group
  48. name: system:serviceaccounts
  49. - apiGroup: rbac.authorization.k8s.io
  50. kind: Group
  51. name: system:authenticated
  52. ---
  53. apiVersion: v1
  54. kind: Namespace
  55. metadata:
  56. name: cattle-system
  57. ---
  58. apiVersion: rbac.authorization.k8s.io/v1
  59. kind: Role
  60. metadata:
  61. name: default-psp-role
  62. namespace: cattle-system
  63. rules:
  64. - apiGroups:
  65. - extensions
  66. resourceNames:
  67. - default-psp
  68. resources:
  69. - podsecuritypolicies
  70. verbs:
  71. - use
  72. ---
  73. apiVersion: rbac.authorization.k8s.io/v1
  74. kind: RoleBinding
  75. metadata:
  76. name: default-psp-rolebinding
  77. namespace: cattle-system
  78. roleRef:
  79. apiGroup: rbac.authorization.k8s.io
  80. kind: Role
  81. name: default-psp-role
  82. subjects:
  83. - apiGroup: rbac.authorization.k8s.io
  84. kind: Group
  85. name: system:serviceaccounts
  86. - apiGroup: rbac.authorization.k8s.io
  87. kind: Group
  88. name: system:authenticated
  89. ---
  90. apiVersion: policy/v1beta1
  91. kind: PodSecurityPolicy
  92. metadata:
  93. name: restricted
  94. spec:
  95. requiredDropCapabilities:
  96. - NET_RAW
  97. privileged: false
  98. allowPrivilegeEscalation: false
  99. defaultAllowPrivilegeEscalation: false
  100. fsGroup:
  101. rule: RunAsAny
  102. runAsUser:
  103. rule: MustRunAsNonRoot
  104. seLinux:
  105. rule: RunAsAny
  106. supplementalGroups:
  107. rule: RunAsAny
  108. volumes:
  109. - emptyDir
  110. - secret
  111. - persistentVolumeClaim
  112. - downwardAPI
  113. - configMap
  114. - projected
  115. ---
  116. apiVersion: rbac.authorization.k8s.io/v1
  117. kind: ClusterRole
  118. metadata:
  119. name: psp:restricted
  120. rules:
  121. - apiGroups:
  122. - extensions
  123. resourceNames:
  124. - restricted
  125. resources:
  126. - podsecuritypolicies
  127. verbs:
  128. - use
  129. ---
  130. apiVersion: rbac.authorization.k8s.io/v1
  131. kind: ClusterRoleBinding
  132. metadata:
  133. name: psp:restricted
  134. roleRef:
  135. apiGroup: rbac.authorization.k8s.io
  136. kind: ClusterRole
  137. name: psp:restricted
  138. subjects:
  139. - apiGroup: rbac.authorization.k8s.io
  140. kind: Group
  141. name: system:serviceaccounts
  142. - apiGroup: rbac.authorization.k8s.io
  143. kind: Group
  144. name: system:authenticated
  145. ---
  146. apiVersion: v1
  147. kind: ServiceAccount
  148. metadata:
  149. name: tiller
  150. namespace: kube-system
  151. ---
  152. apiVersion: rbac.authorization.k8s.io/v1
  153. kind: ClusterRoleBinding
  154. metadata:
  155. name: tiller
  156. roleRef:
  157. apiGroup: rbac.authorization.k8s.io
  158. kind: ClusterRole
  159. name: cluster-admin
  160. subjects:
  161. - kind: ServiceAccount
  162. name: tiller
  163. namespace: kube-system
  164. ignore_docker_version: true
  165. kubernetes_version: v1.15.9-rancher1-1
  166. #
  167. # If you are using calico on AWS
  168. #
  169. # network:
  170. # plugin: calico
  171. # calico_network_provider:
  172. # cloud_provider: aws
  173. #
  174. # # To specify flannel interface
  175. #
  176. # network:
  177. # plugin: flannel
  178. # flannel_network_provider:
  179. # iface: eth1
  180. #
  181. # # To specify flannel interface for canal plugin
  182. #
  183. # network:
  184. # plugin: canal
  185. # canal_network_provider:
  186. # iface: eth1
  187. #
  188. network:
  189. mtu: 0
  190. plugin: canal
  191. #
  192. # services:
  193. # kube-api:
  194. # service_cluster_ip_range: 10.43.0.0/16
  195. # kube-controller:
  196. # cluster_cidr: 10.42.0.0/16
  197. # service_cluster_ip_range: 10.43.0.0/16
  198. # kubelet:
  199. # cluster_domain: cluster.local
  200. # cluster_dns_server: 10.43.0.10
  201. #
  202. services:
  203. etcd:
  204. backup_config:
  205. enabled: false
  206. interval_hours: 12
  207. retention: 6
  208. safe_timestamp: false
  209. creation: 12h
  210. extra_args:
  211. election-timeout: "5000"
  212. heartbeat-interval: "500"
  213. gid: 52034
  214. retention: 72h
  215. snapshot: false
  216. uid: 52034
  217. kube_api:
  218. always_pull_images: false
  219. audit_log:
  220. enabled: true
  221. event_rate_limit:
  222. enabled: true
  223. pod_security_policy: true
  224. secrets_encryption_config:
  225. enabled: true
  226. service_node_port_range: 30000-32767
  227. kube_controller:
  228. extra_args:
  229. address: 127.0.0.1
  230. feature-gates: RotateKubeletServerCertificate=true
  231. profiling: "false"
  232. terminated-pod-gc-threshold: "1000"
  233. kubelet:
  234. extra_args:
  235. anonymous-auth: "false"
  236. event-qps: "0"
  237. feature-gates: RotateKubeletServerCertificate=true
  238. make-iptables-util-chains: "true"
  239. protect-kernel-defaults: "true"
  240. streaming-connection-idle-timeout: 1800s
  241. tls-cipher-suites: >-
  242. TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
  243. fail_swap_on: false
  244. generate_serving_certificate: true
  245. scheduler:
  246. extra_args:
  247. address: 127.0.0.1
  248. profiling: "false"
  249. ssh_agent_auth: false
  250. windows_prefered_cluster: false

安全加固的 Ubuntu 18.04 LTS cloud-config参考配置

这个供您参考的cloud-config通常被用于云基础架构环境中,来进行计算实例的配置管理。这个参考配置了在安装 kubernetes 之前需要的 Ubuntu 操作系统级别的设置。

  1. #cloud-config
  2. packages:
  3. - curl
  4. - jq
  5. runcmd:
  6. - sysctl -w vm.overcommit_memory=1
  7. - sysctl -w kernel.panic=10
  8. - sysctl -w kernel.panic_on_oops=1
  9. - curl https://releases.rancher.com/install-docker/18.09.sh | sh
  10. - usermod -aG docker ubuntu
  11. - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done
  12. - addgroup --gid 52034 etcd
  13. - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
  14. write_files:
  15. - path: /etc/sysctl.d/kubelet.conf
  16. owner: root:root
  17. permissions: "0644"
  18. content: |
  19. vm.overcommit_memory=1
  20. kernel.panic=10
  21. kernel.panic_on_oops=1