本文讲解了如何使您的集群符合互联网安全中心发布的 Kubernetes 安全基准,保护集群中节点的安全。安装 Kubernetes 之前,请按照本指南进行操作。加固指南旨在与特定版本的 CIS Kubernetes Benchmark,Kubernetes 和 Rancher 一起使用。

加固指南旨在与特定版本的 CIS Kubernetes Benchmark,Kubernetes 和 Rancher 一起使用:

加固指南版本Rancher 版本CIS Benchmark 版本Kubernetes 版本
加固指南 v2.4.0Rancher v2.4.0Benchmark v1.5Kubernetes v1.15

点击这里下载 PDF 版本的加固指南

概览

下面的安全加固指南是针对在生产环境的 Rancher v2.4.0 中使用 Kubernetes v1.15 版本的集群。它概述了如何满足互联网安全中心(CIS)提出的 Kubernetes 安全标准。

有关如果根据官方 CIS 基准评估集群的更多详细信息,请参阅CIS Benchmark Rancher 自测指南 - Rancher v2.4.0

已知问题

如果注册自定义节点时只提供了公共 IP,在 CIS 1.5 加固设置中,将无法正常在 Rancher UI 中使用执行命令行查看日志功能。

  • 如果注册自定义节点时只提供了公共 IP,在 CIS 1.5 加固设置中,将无法正常在 Rancher UI 中使用执行命令行查看日志功能。如果想要使用上述两个功能,请在注册自定义节点时提供私有 IP 地址。
  • default_pod_security_policy_template_id:restricted时,Rancher 在默认的 service account 中创建角色绑定集群角色绑定。CIS 1.5 要求默认 service account 没有绑定任何角色,不提供 service account 的 token,不分配特定的权限。

配置内核运行时参数

对于集群中的所有类型的节点,都建议使用以下的sysctl配置。在/etc/sysctl.d/90-kubelet.conf中设置以下参数:

  1. vm.overcommit_memory=1
  2. vm.panic_on_oom=0
  3. kernel.panic=10
  4. kernel.panic_on_oops=1
  5. kernel.keys.root_maxbytes=25000000

执行sysctl -p /etc/sysctl.d/90-kubelet.conf来启用配置。

配置etcd用户和组

在安装 RKE 之前,需要设置etcd服务的用户帐户和组。etcd用户的uidgid将在 RKE 的config.yml中使用,以在安装期间为文件和目录设置适当的权限。

以下命令使用52034作为uidgid的例子。任何有效的未使用的uidgid也可以用来代替52034

创建etcd用户和组

要创建etcd组,请运行以下控制台命令。

  1. addgroup --gid 52034 etcd
  2. useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd

使用etcd用户的uidgid更新 RKE 的 config.yml文件:

  1. services:
  2. etcd:
  3. gid: 52034
  4. uid: 52034

default服务账号的automountServiceAccountToken设置为 false。

Kubernetes 提供了一个默认服务账号(Service Account),如果集群的工作负载中没有为 Pod 分配任何特定服务账号,那么它将会使用这个default的服务账号。在需要从 Pod 访问 Kubernetes API 的情况下,应为该 Pod 创建一个特定的服务账号,并向该服务账号授予权限。这个default的服务账户应该被设置为不提供服务账号令牌(service account token)和任何权限。将automountServiceAccountToken设置为 false 之后,Kubernetes 在启动 Pod 时,将不会自动注入default服务账户。

对于标准 RKE 安装中包括defaultkube-system在内的每个命名空间,default服务账户必须包含这个值。

  1. automountServiceAccountToken: false

把下面的 yaml 另存为account_update.yaml

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4. name: default
  5. automountServiceAccountToken: false

创建一个名称为account_update.sh的脚本。通过运行chmod +x account_update.sh,使这个脚本有执行权限。

  1. #!/bin/bash -e
  2. for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
  3. kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
  4. done

确保所有命名空间均已定义网络策略

在同一个 Kubernetes 集群上运行不同的应用程序会产生一个风险,那就是应用可能受到相邻应用程序的攻击。为了确保容器只能与预期的容器进行通信,网络细分是必不可少的。通过设置网络策略(Network Policy),可以设置哪些 Pod 之间可以通信,以及是否可以和其他网络端点进行通信。

网络策略是作用于命名空间范围的。将网络策略应用于给定命名空间时,所有不被这个策略允许的流量将被拒绝。然而,如果命名空间中没有设置网络策略,那么进出这个命名空间中 Pod 的所有流量都将被允许。要使用网络策略,必须启用 CNI(容器网络接口)插件。本指南使用canal提供策略实施。您可以在这里找到有关 CNI 插件的其他信息。

在集群上启用 CN​​I 插件后,您可以设置一个默认的网络策略。下面是一个宽松的网络策略示例,仅供参考。如果您想要允许到某个命名空间内所有 Pod 的流量(即使已经添加了一些策略,使得一些 Pods 被隔离了),您可以创建一个策略,明确允许该命名空间中的所有流量。将以下yaml另存为 default-allow-all.yaml。额外关于网络策略的信息,请查看Kubernetes 官方文档

:::important 重要 这个NetworkPolicy示例不建议在生产环境中使用。 :::

  1. ---
  2. apiVersion: networking.k8s.io/v1
  3. kind: NetworkPolicy
  4. metadata:
  5. name: default-allow-all
  6. spec:
  7. podSelector: {}
  8. ingress:
  9. - {}
  10. egress:
  11. - {}
  12. policyTypes:
  13. - Ingress
  14. - Egress

创建一个名称为apply_networkPolicy_to_all_ns.sh的脚本。通过运行chmod +x apply_networkPolicy_to_all_ns.sh,使这个脚本有执行权限

  1. #!/bin/bash -e
  2. for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
  3. kubectl apply -f default-allow-all.yaml -n ${namespace}
  4. done

运行脚本,以使全部的命名空间使用这个default-allow-all.yaml文件中的宽松NetworkPolicy

加固的 RKE cluster.yml 配置参考

您可以用这个供您参考的cluster.yml,通过 RKE CLI 来创建安全加固的 Rancher Kubernetes Engine(RKE)集群。有关每个配置的详细信息,请参阅RKE 文档。这个cluster.yml问号不包括所需的nodes指令,它将根据你的环境而变化。有关如何节点配置的文档可以参考RKE 节点配置示例

  1. # If you intend to deploy Kubernetes in an air-gapped environment,
  2. # please consult the documentation on how to configure custom RKE images.
  3. kubernetes_version: "v1.15.9-rancher1-1"
  4. enable_network_policy: true
  5. default_pod_security_policy_template_id: "restricted"
  6. # the nodes directive is required and will vary depending on your environment
  7. # documentation for node configuration can be found here:
  8. # https://docs.rancher.cn/docs/rke/config-options/nodes/_index
  9. services:
  10. etcd:
  11. uid: 52034
  12. gid: 52034
  13. kube-api:
  14. pod_security_policy: true
  15. secrets_encryption_config:
  16. enabled: true
  17. audit_log:
  18. enabled: true
  19. admission_configuration:
  20. event_rate_limit:
  21. enabled: true
  22. kube-controller:
  23. extra_args:
  24. feature-gates: "RotateKubeletServerCertificate=true"
  25. scheduler:
  26. image: ""
  27. extra_args: {}
  28. extra_binds: []
  29. extra_env: []
  30. kubelet:
  31. generate_serving_certificate: true
  32. extra_args:
  33. feature-gates: "RotateKubeletServerCertificate=true"
  34. protect-kernel-defaults: "true"
  35. tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
  36. extra_binds: []
  37. extra_env: []
  38. cluster_domain: ""
  39. infra_container_image: ""
  40. cluster_dns_server: ""
  41. fail_swap_on: false
  42. kubeproxy:
  43. image: ""
  44. extra_args: {}
  45. extra_binds: []
  46. extra_env: []
  47. network:
  48. plugin: ""
  49. options: {}
  50. mtu: 0
  51. node_selector: {}
  52. authentication:
  53. strategy: ""
  54. sans: []
  55. webhook: null
  56. addons: |
  57. ---
  58. apiVersion: v1
  59. kind: Namespace
  60. metadata:
  61. name: ingress-nginx
  62. ---
  63. apiVersion: rbac.authorization.k8s.io/v1
  64. kind: Role
  65. metadata:
  66. name: default-psp-role
  67. namespace: ingress-nginx
  68. rules:
  69. - apiGroups:
  70. - extensions
  71. resourceNames:
  72. - default-psp
  73. resources:
  74. - podsecuritypolicies
  75. verbs:
  76. - use
  77. ---
  78. apiVersion: rbac.authorization.k8s.io/v1
  79. kind: RoleBinding
  80. metadata:
  81. name: default-psp-rolebinding
  82. namespace: ingress-nginx
  83. roleRef:
  84. apiGroup: rbac.authorization.k8s.io
  85. kind: Role
  86. name: default-psp-role
  87. subjects:
  88. - apiGroup: rbac.authorization.k8s.io
  89. kind: Group
  90. name: system:serviceaccounts
  91. - apiGroup: rbac.authorization.k8s.io
  92. kind: Group
  93. name: system:authenticated
  94. ---
  95. apiVersion: v1
  96. kind: Namespace
  97. metadata:
  98. name: cattle-system
  99. ---
  100. apiVersion: rbac.authorization.k8s.io/v1
  101. kind: Role
  102. metadata:
  103. name: default-psp-role
  104. namespace: cattle-system
  105. rules:
  106. - apiGroups:
  107. - extensions
  108. resourceNames:
  109. - default-psp
  110. resources:
  111. - podsecuritypolicies
  112. verbs:
  113. - use
  114. ---
  115. apiVersion: rbac.authorization.k8s.io/v1
  116. kind: RoleBinding
  117. metadata:
  118. name: default-psp-rolebinding
  119. namespace: cattle-system
  120. roleRef:
  121. apiGroup: rbac.authorization.k8s.io
  122. kind: Role
  123. name: default-psp-role
  124. subjects:
  125. - apiGroup: rbac.authorization.k8s.io
  126. kind: Group
  127. name: system:serviceaccounts
  128. - apiGroup: rbac.authorization.k8s.io
  129. kind: Group
  130. name: system:authenticated
  131. ---
  132. apiVersion: policy/v1beta1
  133. kind: PodSecurityPolicy
  134. metadata:
  135. name: restricted
  136. spec:
  137. requiredDropCapabilities:
  138. - NET_RAW
  139. privileged: false
  140. allowPrivilegeEscalation: false
  141. defaultAllowPrivilegeEscalation: false
  142. fsGroup:
  143. rule: RunAsAny
  144. runAsUser:
  145. rule: MustRunAsNonRoot
  146. seLinux:
  147. rule: RunAsAny
  148. supplementalGroups:
  149. rule: RunAsAny
  150. volumes:
  151. - emptyDir
  152. - secret
  153. - persistentVolumeClaim
  154. - downwardAPI
  155. - configMap
  156. - projected
  157. ---
  158. apiVersion: rbac.authorization.k8s.io/v1
  159. kind: ClusterRole
  160. metadata:
  161. name: psp:restricted
  162. rules:
  163. - apiGroups:
  164. - extensions
  165. resourceNames:
  166. - restricted
  167. resources:
  168. - podsecuritypolicies
  169. verbs:
  170. - use
  171. ---
  172. apiVersion: rbac.authorization.k8s.io/v1
  173. kind: ClusterRoleBinding
  174. metadata:
  175. name: psp:restricted
  176. roleRef:
  177. apiGroup: rbac.authorization.k8s.io
  178. kind: ClusterRole
  179. name: psp:restricted
  180. subjects:
  181. - apiGroup: rbac.authorization.k8s.io
  182. kind: Group
  183. name: system:serviceaccounts
  184. - apiGroup: rbac.authorization.k8s.io
  185. kind: Group
  186. name: system:authenticated
  187. ---
  188. apiVersion: v1
  189. kind: ServiceAccount
  190. metadata:
  191. name: tiller
  192. namespace: kube-system
  193. ---
  194. apiVersion: rbac.authorization.k8s.io/v1
  195. kind: ClusterRoleBinding
  196. metadata:
  197. name: tiller
  198. roleRef:
  199. apiGroup: rbac.authorization.k8s.io
  200. kind: ClusterRole
  201. name: cluster-admin
  202. subjects:
  203. - kind: ServiceAccount
  204. name: tiller
  205. namespace: kube-system
  206. addons_include: []
  207. system_images:
  208. etcd: ""
  209. alpine: ""
  210. nginx_proxy: ""
  211. cert_downloader: ""
  212. kubernetes_services_sidecar: ""
  213. kubedns: ""
  214. dnsmasq: ""
  215. kubedns_sidecar: ""
  216. kubedns_autoscaler: ""
  217. coredns: ""
  218. coredns_autoscaler: ""
  219. kubernetes: ""
  220. flannel: ""
  221. flannel_cni: ""
  222. calico_node: ""
  223. calico_cni: ""
  224. calico_controllers: ""
  225. calico_ctl: ""
  226. calico_flexvol: ""
  227. canal_node: ""
  228. canal_cni: ""
  229. canal_flannel: ""
  230. canal_flexvol: ""
  231. weave_node: ""
  232. weave_cni: ""
  233. pod_infra_container: ""
  234. ingress: ""
  235. ingress_backend: ""
  236. metrics_server: ""
  237. windows_pod_infra_container: ""
  238. ssh_key_path: ""
  239. ssh_cert_path: ""
  240. ssh_agent_auth: false
  241. authorization:
  242. mode: ""
  243. options: {}
  244. ignore_docker_version: false
  245. private_registries: []
  246. ingress:
  247. provider: ""
  248. options: {}
  249. node_selector: {}
  250. extra_args: {}
  251. dns_policy: ""
  252. extra_envs: []
  253. extra_volumes: []
  254. extra_volume_mounts: []
  255. cluster_name: ""
  256. prefix_path: ""
  257. addon_job_timeout: 0
  258. bastion_host:
  259. address: ""
  260. port: ""
  261. user: ""
  262. ssh_key: ""
  263. ssh_key_path: ""
  264. ssh_cert: ""
  265. ssh_cert_path: ""
  266. monitoring:
  267. provider: ""
  268. options: {}
  269. node_selector: {}
  270. restore:
  271. restore: false
  272. snapshot_name: ""
  273. dns: null

安全加固的 RKE 模板配置参考

这个 RKE 参考模板提供了安装安全加固的 Kubenetes 所需的配置。RKE 模板用于配置 Kubernetes 和定义 Rancher 设置。请参阅Rancher 文档获得更多安装和 RKE 模板的详细信息。

  1. #
  2. # Cluster Config
  3. #
  4. default_pod_security_policy_template_id: restricted
  5. docker_root_dir: /var/lib/docker
  6. enable_cluster_alerting: false
  7. enable_cluster_monitoring: false
  8. enable_network_policy: true
  9. #
  10. # Rancher Config
  11. #
  12. rancher_kubernetes_engine_config:
  13. addon_job_timeout: 30
  14. addons: |-
  15. ---
  16. apiVersion: v1
  17. kind: Namespace
  18. metadata:
  19. name: ingress-nginx
  20. ---
  21. apiVersion: rbac.authorization.k8s.io/v1
  22. kind: Role
  23. metadata:
  24. name: default-psp-role
  25. namespace: ingress-nginx
  26. rules:
  27. - apiGroups:
  28. - extensions
  29. resourceNames:
  30. - default-psp
  31. resources:
  32. - podsecuritypolicies
  33. verbs:
  34. - use
  35. ---
  36. apiVersion: rbac.authorization.k8s.io/v1
  37. kind: RoleBinding
  38. metadata:
  39. name: default-psp-rolebinding
  40. namespace: ingress-nginx
  41. roleRef:
  42. apiGroup: rbac.authorization.k8s.io
  43. kind: Role
  44. name: default-psp-role
  45. subjects:
  46. - apiGroup: rbac.authorization.k8s.io
  47. kind: Group
  48. name: system:serviceaccounts
  49. - apiGroup: rbac.authorization.k8s.io
  50. kind: Group
  51. name: system:authenticated
  52. ---
  53. apiVersion: v1
  54. kind: Namespace
  55. metadata:
  56. name: cattle-system
  57. ---
  58. apiVersion: rbac.authorization.k8s.io/v1
  59. kind: Role
  60. metadata:
  61. name: default-psp-role
  62. namespace: cattle-system
  63. rules:
  64. - apiGroups:
  65. - extensions
  66. resourceNames:
  67. - default-psp
  68. resources:
  69. - podsecuritypolicies
  70. verbs:
  71. - use
  72. ---
  73. apiVersion: rbac.authorization.k8s.io/v1
  74. kind: RoleBinding
  75. metadata:
  76. name: default-psp-rolebinding
  77. namespace: cattle-system
  78. roleRef:
  79. apiGroup: rbac.authorization.k8s.io
  80. kind: Role
  81. name: default-psp-role
  82. subjects:
  83. - apiGroup: rbac.authorization.k8s.io
  84. kind: Group
  85. name: system:serviceaccounts
  86. - apiGroup: rbac.authorization.k8s.io
  87. kind: Group
  88. name: system:authenticated
  89. ---
  90. apiVersion: policy/v1beta1
  91. kind: PodSecurityPolicy
  92. metadata:
  93. name: restricted
  94. spec:
  95. requiredDropCapabilities:
  96. - NET_RAW
  97. privileged: false
  98. allowPrivilegeEscalation: false
  99. defaultAllowPrivilegeEscalation: false
  100. fsGroup:
  101. rule: RunAsAny
  102. runAsUser:
  103. rule: MustRunAsNonRoot
  104. seLinux:
  105. rule: RunAsAny
  106. supplementalGroups:
  107. rule: RunAsAny
  108. volumes:
  109. - emptyDir
  110. - secret
  111. - persistentVolumeClaim
  112. - downwardAPI
  113. - configMap
  114. - projected
  115. ---
  116. apiVersion: rbac.authorization.k8s.io/v1
  117. kind: ClusterRole
  118. metadata:
  119. name: psp:restricted
  120. rules:
  121. - apiGroups:
  122. - extensions
  123. resourceNames:
  124. - restricted
  125. resources:
  126. - podsecuritypolicies
  127. verbs:
  128. - use
  129. ---
  130. apiVersion: rbac.authorization.k8s.io/v1
  131. kind: ClusterRoleBinding
  132. metadata:
  133. name: psp:restricted
  134. roleRef:
  135. apiGroup: rbac.authorization.k8s.io
  136. kind: ClusterRole
  137. name: psp:restricted
  138. subjects:
  139. - apiGroup: rbac.authorization.k8s.io
  140. kind: Group
  141. name: system:serviceaccounts
  142. - apiGroup: rbac.authorization.k8s.io
  143. kind: Group
  144. name: system:authenticated
  145. ---
  146. apiVersion: v1
  147. kind: ServiceAccount
  148. metadata:
  149. name: tiller
  150. namespace: kube-system
  151. ---
  152. apiVersion: rbac.authorization.k8s.io/v1
  153. kind: ClusterRoleBinding
  154. metadata:
  155. name: tiller
  156. roleRef:
  157. apiGroup: rbac.authorization.k8s.io
  158. kind: ClusterRole
  159. name: cluster-admin
  160. subjects:
  161. - kind: ServiceAccount
  162. name: tiller
  163. namespace: kube-system
  164. ignore_docker_version: true
  165. kubernetes_version: v1.15.9-rancher1-1
  166. #
  167. # If you are using calico on AWS
  168. #
  169. # network:
  170. # plugin: calico
  171. # calico_network_provider:
  172. # cloud_provider: aws
  173. #
  174. # # To specify flannel interface
  175. #
  176. # network:
  177. # plugin: flannel
  178. # flannel_network_provider:
  179. # iface: eth1
  180. #
  181. # # To specify flannel interface for canal plugin
  182. #
  183. # network:
  184. # plugin: canal
  185. # canal_network_provider:
  186. # iface: eth1
  187. #
  188. network:
  189. mtu: 0
  190. plugin: canal
  191. #
  192. # services:
  193. # kube-api:
  194. # service_cluster_ip_range: 10.43.0.0/16
  195. # kube-controller:
  196. # cluster_cidr: 10.42.0.0/16
  197. # service_cluster_ip_range: 10.43.0.0/16
  198. # kubelet:
  199. # cluster_domain: cluster.local
  200. # cluster_dns_server: 10.43.0.10
  201. #
  202. services:
  203. etcd:
  204. backup_config:
  205. enabled: false
  206. interval_hours: 12
  207. retention: 6
  208. safe_timestamp: false
  209. creation: 12h
  210. extra_args:
  211. election-timeout: "5000"
  212. heartbeat-interval: "500"
  213. gid: 52034
  214. retention: 72h
  215. snapshot: false
  216. uid: 52034
  217. kube_api:
  218. always_pull_images: false
  219. audit_log:
  220. enabled: true
  221. event_rate_limit:
  222. enabled: true
  223. pod_security_policy: true
  224. secrets_encryption_config:
  225. enabled: true
  226. service_node_port_range: 30000-32767
  227. kube_controller:
  228. extra_args:
  229. address: 127.0.0.1
  230. feature-gates: RotateKubeletServerCertificate=true
  231. profiling: "false"
  232. terminated-pod-gc-threshold: "1000"
  233. kubelet:
  234. extra_args:
  235. anonymous-auth: "false"
  236. event-qps: "0"
  237. feature-gates: RotateKubeletServerCertificate=true
  238. make-iptables-util-chains: "true"
  239. protect-kernel-defaults: "true"
  240. streaming-connection-idle-timeout: 1800s
  241. tls-cipher-suites: >-
  242. TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
  243. fail_swap_on: false
  244. generate_serving_certificate: true
  245. scheduler:
  246. extra_args:
  247. address: 127.0.0.1
  248. profiling: "false"
  249. ssh_agent_auth: false
  250. windows_prefered_cluster: false

安全加固的 Ubuntu 18.04 LTS cloud-config参考配置

这个供您参考的cloud-config通常被用于云基础架构环境中,来进行计算实例的配置管理。这个参考配置了在安装 kubernetes 之前需要的 Ubuntu 操作系统级别的设置。

  1. #cloud-config
  2. packages:
  3. - curl
  4. - jq
  5. runcmd:
  6. - sysctl -w vm.overcommit_memory=1
  7. - sysctl -w kernel.panic=10
  8. - sysctl -w kernel.panic_on_oops=1
  9. - curl https://releases.rancher.com/install-docker/18.09.sh | sh
  10. - usermod -aG docker ubuntu
  11. - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done
  12. - addgroup --gid 52034 etcd
  13. - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
  14. write_files:
  15. - path: /etc/sysctl.d/kubelet.conf
  16. owner: root:root
  17. permissions: "0644"
  18. content: |
  19. vm.overcommit_memory=1
  20. kernel.panic=10
  21. kernel.panic_on_oops=1