Ceph Authorization

Ceph is a highly scalable distributed storage solution that uniquely delivers object, block, and file storage in one unified system. You can enforce fine-grained authorization over Ceph’s Object Storage using OPA. Ceph’s Object Storage essentially consists of a Ceph Storage Cluster and a Ceph Object Gateway.

The Ceph Object Gateway is an object storage interface built on top of librados to provide applications with a RESTful gateway to Ceph Storage Clusters.

OPA is integrated with the Ceph Object Gateway daemon (RGW), which is an HTTP server that interacts with a Ceph Storage Cluster and provides interfaces compatible with OpenStack Swift and Amazon S3.

When the Ceph Object Gateway gets a request, it checks with OPA whether the request should be allowed or not. OPA makes a decision (allow or deny) based on the policies and data it has access to and sends the decision back to the Ceph Object Gateway for enforcement.

Goals

This tutorial shows how to enforce custom policies over the S3 API to the Ceph Storage Cluster which applications use to put and get data.

This tutorial uses Rook to run Ceph inside a Kubernetes cluster.

Prerequisites

This tutorial requires Kubernetes 1.9 or later. To run the tutorial locally, we recommend using minikube in version v0.28+ with Kubernetes 1.10 (which is the default).

Steps

1. Start Minikube

  1. minikube start

2. Deploy the Rook Operator

Deploy the Rook system components, which include the Rook agent and Rook operator pods.

Save the operator spec as operator.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4. name: rook-ceph-system
  5. ---
  6. apiVersion: apiextensions.k8s.io/v1beta1
  7. kind: CustomResourceDefinition
  8. metadata:
  9. name: cephclusters.ceph.rook.io
  10. spec:
  11. group: ceph.rook.io
  12. names:
  13. kind: CephCluster
  14. listKind: CephClusterList
  15. plural: cephclusters
  16. singular: cephcluster
  17. scope: Namespaced
  18. version: v1
  19. validation:
  20. openAPIV3Schema:
  21. properties:
  22. spec:
  23. properties:
  24. cephVersion:
  25. properties:
  26. allowUnsupported:
  27. type: boolean
  28. image:
  29. type: string
  30. name:
  31. pattern: ^(luminous|mimic|nautilus)$
  32. type: string
  33. dashboard:
  34. properties:
  35. enabled:
  36. type: boolean
  37. urlPrefix:
  38. type: string
  39. port:
  40. type: integer
  41. dataDirHostPath:
  42. pattern: ^/(\S+)
  43. type: string
  44. mon:
  45. properties:
  46. allowMultiplePerNode:
  47. type: boolean
  48. count:
  49. maximum: 9
  50. minimum: 1
  51. type: integer
  52. required:
  53. - count
  54. network:
  55. properties:
  56. hostNetwork:
  57. type: boolean
  58. storage:
  59. properties:
  60. nodes:
  61. items: {}
  62. type: array
  63. useAllDevices: {}
  64. useAllNodes:
  65. type: boolean
  66. required:
  67. - mon
  68. additionalPrinterColumns:
  69. - name: DataDirHostPath
  70. type: string
  71. description: Directory used on the K8s nodes
  72. JSONPath: .spec.dataDirHostPath
  73. - name: MonCount
  74. type: string
  75. description: Number of MONs
  76. JSONPath: .spec.mon.count
  77. - name: Age
  78. type: date
  79. JSONPath: .metadata.creationTimestamp
  80. - name: State
  81. type: string
  82. description: Current State
  83. JSONPath: .status.state
  84. ---
  85. apiVersion: apiextensions.k8s.io/v1beta1
  86. kind: CustomResourceDefinition
  87. metadata:
  88. name: cephfilesystems.ceph.rook.io
  89. spec:
  90. group: ceph.rook.io
  91. names:
  92. kind: CephFilesystem
  93. listKind: CephFilesystemList
  94. plural: cephfilesystems
  95. singular: cephfilesystem
  96. scope: Namespaced
  97. version: v1
  98. additionalPrinterColumns:
  99. - name: MdsCount
  100. type: string
  101. description: Number of MDSs
  102. JSONPath: .spec.metadataServer.activeCount
  103. - name: Age
  104. type: date
  105. JSONPath: .metadata.creationTimestamp
  106. ---
  107. apiVersion: apiextensions.k8s.io/v1beta1
  108. kind: CustomResourceDefinition
  109. metadata:
  110. name: cephobjectstores.ceph.rook.io
  111. spec:
  112. group: ceph.rook.io
  113. names:
  114. kind: CephObjectStore
  115. listKind: CephObjectStoreList
  116. plural: cephobjectstores
  117. singular: cephobjectstore
  118. scope: Namespaced
  119. version: v1
  120. ---
  121. apiVersion: apiextensions.k8s.io/v1beta1
  122. kind: CustomResourceDefinition
  123. metadata:
  124. name: cephobjectstoreusers.ceph.rook.io
  125. spec:
  126. group: ceph.rook.io
  127. names:
  128. kind: CephObjectStoreUser
  129. listKind: CephObjectStoreUserList
  130. plural: cephobjectstoreusers
  131. singular: cephobjectstoreuser
  132. scope: Namespaced
  133. version: v1
  134. ---
  135. apiVersion: apiextensions.k8s.io/v1beta1
  136. kind: CustomResourceDefinition
  137. metadata:
  138. name: cephblockpools.ceph.rook.io
  139. spec:
  140. group: ceph.rook.io
  141. names:
  142. kind: CephBlockPool
  143. listKind: CephBlockPoolList
  144. plural: cephblockpools
  145. singular: cephblockpool
  146. scope: Namespaced
  147. version: v1
  148. ---
  149. apiVersion: apiextensions.k8s.io/v1beta1
  150. kind: CustomResourceDefinition
  151. metadata:
  152. name: volumes.rook.io
  153. spec:
  154. group: rook.io
  155. names:
  156. kind: Volume
  157. listKind: VolumeList
  158. plural: volumes
  159. singular: volume
  160. shortNames:
  161. - rv
  162. scope: Namespaced
  163. version: v1alpha2
  164. ---
  165. apiVersion: rbac.authorization.k8s.io/v1beta1
  166. kind: ClusterRole
  167. metadata:
  168. name: rook-ceph-cluster-mgmt
  169. labels:
  170. operator: rook
  171. storage-backend: ceph
  172. rules:
  173. - apiGroups:
  174. - ""
  175. resources:
  176. - secrets
  177. - pods
  178. - pods/log
  179. - services
  180. - configmaps
  181. verbs:
  182. - get
  183. - list
  184. - watch
  185. - patch
  186. - create
  187. - update
  188. - delete
  189. - apiGroups:
  190. - extensions
  191. resources:
  192. - deployments
  193. - daemonsets
  194. - replicasets
  195. verbs:
  196. - get
  197. - list
  198. - watch
  199. - create
  200. - update
  201. - delete
  202. ---
  203. apiVersion: rbac.authorization.k8s.io/v1beta1
  204. kind: Role
  205. metadata:
  206. name: rook-ceph-system
  207. namespace: rook-ceph-system
  208. labels:
  209. operator: rook
  210. storage-backend: ceph
  211. rules:
  212. - apiGroups:
  213. - ""
  214. resources:
  215. - pods
  216. - configmaps
  217. verbs:
  218. - get
  219. - list
  220. - watch
  221. - patch
  222. - create
  223. - update
  224. - delete
  225. - apiGroups:
  226. - extensions
  227. resources:
  228. - daemonsets
  229. verbs:
  230. - get
  231. - list
  232. - watch
  233. - create
  234. - update
  235. - delete
  236. ---
  237. apiVersion: rbac.authorization.k8s.io/v1beta1
  238. kind: ClusterRole
  239. metadata:
  240. name: rook-ceph-global
  241. labels:
  242. operator: rook
  243. storage-backend: ceph
  244. rules:
  245. - apiGroups:
  246. - ""
  247. resources:
  248. - pods
  249. - nodes
  250. - nodes/proxy
  251. verbs:
  252. - get
  253. - list
  254. - watch
  255. - apiGroups:
  256. - ""
  257. resources:
  258. - events
  259. - persistentvolumes
  260. - persistentvolumeclaims
  261. verbs:
  262. - get
  263. - list
  264. - watch
  265. - patch
  266. - create
  267. - update
  268. - delete
  269. - apiGroups:
  270. - storage.k8s.io
  271. resources:
  272. - storageclasses
  273. verbs:
  274. - get
  275. - list
  276. - watch
  277. - apiGroups:
  278. - batch
  279. resources:
  280. - jobs
  281. verbs:
  282. - get
  283. - list
  284. - watch
  285. - create
  286. - update
  287. - delete
  288. - apiGroups:
  289. - ceph.rook.io
  290. resources:
  291. - "*"
  292. verbs:
  293. - "*"
  294. - apiGroups:
  295. - rook.io
  296. resources:
  297. - "*"
  298. verbs:
  299. - "*"
  300. ---
  301. kind: ClusterRole
  302. apiVersion: rbac.authorization.k8s.io/v1beta1
  303. metadata:
  304. name: rook-ceph-mgr-cluster
  305. labels:
  306. operator: rook
  307. storage-backend: ceph
  308. rules:
  309. - apiGroups:
  310. - ""
  311. resources:
  312. - configmaps
  313. - nodes
  314. - nodes/proxy
  315. verbs:
  316. - get
  317. - list
  318. - watch
  319. ---
  320. apiVersion: v1
  321. kind: ServiceAccount
  322. metadata:
  323. name: rook-ceph-system
  324. namespace: rook-ceph-system
  325. labels:
  326. operator: rook
  327. storage-backend: ceph
  328. ---
  329. kind: RoleBinding
  330. apiVersion: rbac.authorization.k8s.io/v1beta1
  331. metadata:
  332. name: rook-ceph-system
  333. namespace: rook-ceph-system
  334. labels:
  335. operator: rook
  336. storage-backend: ceph
  337. roleRef:
  338. apiGroup: rbac.authorization.k8s.io
  339. kind: Role
  340. name: rook-ceph-system
  341. subjects:
  342. - kind: ServiceAccount
  343. name: rook-ceph-system
  344. namespace: rook-ceph-system
  345. ---
  346. kind: ClusterRoleBinding
  347. apiVersion: rbac.authorization.k8s.io/v1beta1
  348. metadata:
  349. name: rook-ceph-global
  350. namespace: rook-ceph-system
  351. labels:
  352. operator: rook
  353. storage-backend: ceph
  354. roleRef:
  355. apiGroup: rbac.authorization.k8s.io
  356. kind: ClusterRole
  357. name: rook-ceph-global
  358. subjects:
  359. - kind: ServiceAccount
  360. name: rook-ceph-system
  361. namespace: rook-ceph-system
  362. ---
  363. apiVersion: apps/v1beta1
  364. kind: Deployment
  365. metadata:
  366. name: rook-ceph-operator
  367. namespace: rook-ceph-system
  368. labels:
  369. operator: rook
  370. storage-backend: ceph
  371. spec:
  372. replicas: 1
  373. template:
  374. metadata:
  375. labels:
  376. app: rook-ceph-operator
  377. spec:
  378. serviceAccountName: rook-ceph-system
  379. containers:
  380. - name: rook-ceph-operator
  381. image: openpolicyagent/rook-operator:latest
  382. args: ["ceph", "operator"]
  383. volumeMounts:
  384. - mountPath: /var/lib/rook
  385. name: rook-config
  386. - mountPath: /etc/ceph
  387. name: default-config-dir
  388. env:
  389. - name: ROOK_ALLOW_MULTIPLE_FILESYSTEMS
  390. value: "false"
  391. - name: ROOK_LOG_LEVEL
  392. value: "INFO"
  393. - name: ROOK_MON_HEALTHCHECK_INTERVAL
  394. value: "45s"
  395. - name: ROOK_MON_OUT_TIMEOUT
  396. value: "300s"
  397. - name: ROOK_DISCOVER_DEVICES_INTERVAL
  398. value: "60m"
  399. - name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED
  400. value: "false"
  401. - name: ROOK_ENABLE_SELINUX_RELABELING
  402. value: "true"
  403. - name: NODE_NAME
  404. valueFrom:
  405. fieldRef:
  406. fieldPath: spec.nodeName
  407. - name: POD_NAME
  408. valueFrom:
  409. fieldRef:
  410. fieldPath: metadata.name
  411. - name: POD_NAMESPACE
  412. valueFrom:
  413. fieldRef:
  414. fieldPath: metadata.namespace
  415. volumes:
  416. - name: rook-config
  417. emptyDir: {}
  418. - name: default-config-dir
  419. emptyDir: {}

Create the operator:

  1. kubectl create -f operator.yaml

Verify that rook-ceph-operator, rook-ceph-agent, and rook-discover pods are in the Running state.

  1. kubectl -n rook-ceph-system get pod

The Rook operator image openpolicyagent/rook-operator:latest used in the tutorial supports the latest version of Ceph (nautilus). Rook currently does not support the latest version of Ceph. See this Github issue for more details.

3. Deploy OPA on top of Kubernetes

Save the OPA spec as opa.yaml:

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4. name: rook-ceph
  5. ---
  6. kind: Service
  7. apiVersion: v1
  8. metadata:
  9. name: opa
  10. namespace: rook-ceph
  11. labels:
  12. app: opa
  13. rook_cluster: rook-ceph
  14. spec:
  15. type: NodePort
  16. selector:
  17. app: opa
  18. ports:
  19. - name: http
  20. protocol: TCP
  21. port: 8181
  22. targetPort: 8181
  23. ---
  24. apiVersion: extensions/v1beta1
  25. kind: Deployment
  26. metadata:
  27. labels:
  28. app: opa
  29. namespace: rook-ceph
  30. name: opa
  31. spec:
  32. replicas: 1
  33. selector:
  34. matchLabels:
  35. app: opa
  36. template:
  37. metadata:
  38. labels:
  39. app: opa
  40. name: opa
  41. spec:
  42. containers:
  43. - name: opa
  44. image: openpolicyagent/opa:0.12.2
  45. ports:
  46. - name: http
  47. containerPort: 8181
  48. args:
  49. - "run"
  50. - "--ignore=.*"
  51. - "--server"
  52. - "--log-level=debug"
  53. - "/policies/authz.rego"
  54. volumeMounts:
  55. - readOnly: true
  56. mountPath: /policies
  57. name: authz-policy
  58. volumes:
  59. - name: authz-policy
  60. configMap:
  61. name: authz-policy
  62. ---
  63. kind: ConfigMap
  64. apiVersion: v1
  65. metadata:
  66. name: authz-policy
  67. namespace: rook-ceph
  68. data:
  69. authz.rego: |
  70. package ceph.authz
  71. default allow = false
  72. #-----------------------------------------------------------------------------
  73. # Data structures containing location info about users and buckets.
  74. # In real-world deployments, these data structures could be loaded into
  75. # OPA as raw JSON data. The JSON data could also be pulled from external
  76. # sources like AD, Git, etc.
  77. #-----------------------------------------------------------------------------
  78. # user-location information
  79. user_location = {
  80. "alice": "UK",
  81. "bob": "USA"
  82. }
  83. # bucket-location information
  84. bucket_location = {
  85. "supersecretbucket": "USA"
  86. }
  87. allow {
  88. input.method == "HEAD"
  89. is_user_in_bucket_location(input.user_info.user_id, input.bucket_info.bucket.name)
  90. }
  91. allow {
  92. input.method == "GET"
  93. }
  94. allow {
  95. input.method == "PUT"
  96. input.user_info.display_name == "Bob"
  97. }
  98. allow {
  99. input.method == "DELETE"
  100. input.user_info.display_name == "Bob"
  101. }
  102. # Check if the user and the bucket being accessed belong to the same
  103. # location.
  104. is_user_in_bucket_location(user, bucket) {
  105. user_location[user] == bucket_location[bucket]
  106. }
  1. kubectl apply -f opa.yaml

The OPA spec contains a ConfigMap where an OPA policy has been defined. This policy will be used to authorize requests received by the Ceph Object Gateway. More details on this policy will be covered later in the tutorial.

Verify that the OPA pod is Running.

  1. kubectl -n rook-ceph get pod -l app=opa

4. Create a Ceph Cluster

For the cluster to survive reboots, make sure you set the dataDirHostPath property that is valid for your hosts. For minikube, dataDirHostPath is set to /data/rook.

Save the cluster spec as cluster.yaml:

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4. name: rook-ceph-osd
  5. namespace: rook-ceph
  6. ---
  7. apiVersion: v1
  8. kind: ServiceAccount
  9. metadata:
  10. name: rook-ceph-mgr
  11. namespace: rook-ceph
  12. ---
  13. kind: Role
  14. apiVersion: rbac.authorization.k8s.io/v1beta1
  15. metadata:
  16. name: rook-ceph-osd
  17. namespace: rook-ceph
  18. rules:
  19. - apiGroups: [""]
  20. resources: ["configmaps"]
  21. verbs: [ "get", "list", "watch", "create", "update", "delete" ]
  22. ---
  23. kind: Role
  24. apiVersion: rbac.authorization.k8s.io/v1beta1
  25. metadata:
  26. name: rook-ceph-mgr-system
  27. namespace: rook-ceph
  28. rules:
  29. - apiGroups:
  30. - ""
  31. resources:
  32. - configmaps
  33. verbs:
  34. - get
  35. - list
  36. - watch
  37. ---
  38. kind: Role
  39. apiVersion: rbac.authorization.k8s.io/v1beta1
  40. metadata:
  41. name: rook-ceph-mgr
  42. namespace: rook-ceph
  43. rules:
  44. - apiGroups:
  45. - ""
  46. resources:
  47. - pods
  48. - services
  49. verbs:
  50. - get
  51. - list
  52. - watch
  53. - apiGroups:
  54. - batch
  55. resources:
  56. - jobs
  57. verbs:
  58. - get
  59. - list
  60. - watch
  61. - create
  62. - update
  63. - delete
  64. - apiGroups:
  65. - ceph.rook.io
  66. resources:
  67. - "*"
  68. verbs:
  69. - "*"
  70. ---
  71. kind: RoleBinding
  72. apiVersion: rbac.authorization.k8s.io/v1beta1
  73. metadata:
  74. name: rook-ceph-cluster-mgmt
  75. namespace: rook-ceph
  76. roleRef:
  77. apiGroup: rbac.authorization.k8s.io
  78. kind: ClusterRole
  79. name: rook-ceph-cluster-mgmt
  80. subjects:
  81. - kind: ServiceAccount
  82. name: rook-ceph-system
  83. namespace: rook-ceph-system
  84. ---
  85. kind: RoleBinding
  86. apiVersion: rbac.authorization.k8s.io/v1beta1
  87. metadata:
  88. name: rook-ceph-osd
  89. namespace: rook-ceph
  90. roleRef:
  91. apiGroup: rbac.authorization.k8s.io
  92. kind: Role
  93. name: rook-ceph-osd
  94. subjects:
  95. - kind: ServiceAccount
  96. name: rook-ceph-osd
  97. namespace: rook-ceph
  98. ---
  99. kind: RoleBinding
  100. apiVersion: rbac.authorization.k8s.io/v1beta1
  101. metadata:
  102. name: rook-ceph-mgr
  103. namespace: rook-ceph
  104. roleRef:
  105. apiGroup: rbac.authorization.k8s.io
  106. kind: Role
  107. name: rook-ceph-mgr
  108. subjects:
  109. - kind: ServiceAccount
  110. name: rook-ceph-mgr
  111. namespace: rook-ceph
  112. ---
  113. kind: RoleBinding
  114. apiVersion: rbac.authorization.k8s.io/v1beta1
  115. metadata:
  116. name: rook-ceph-mgr-system
  117. namespace: rook-ceph-system
  118. roleRef:
  119. apiGroup: rbac.authorization.k8s.io
  120. kind: Role
  121. name: rook-ceph-mgr-system
  122. subjects:
  123. - kind: ServiceAccount
  124. name: rook-ceph-mgr
  125. namespace: rook-ceph
  126. ---
  127. kind: RoleBinding
  128. apiVersion: rbac.authorization.k8s.io/v1beta1
  129. metadata:
  130. name: rook-ceph-mgr-cluster
  131. namespace: rook-ceph
  132. roleRef:
  133. apiGroup: rbac.authorization.k8s.io
  134. kind: ClusterRole
  135. name: rook-ceph-mgr-cluster
  136. subjects:
  137. - kind: ServiceAccount
  138. name: rook-ceph-mgr
  139. namespace: rook-ceph
  140. ---
  141. apiVersion: ceph.rook.io/v1
  142. kind: CephCluster
  143. metadata:
  144. name: rook-ceph
  145. namespace: rook-ceph
  146. spec:
  147. cephVersion:
  148. image: ceph/daemon-base:latest-master
  149. allowUnsupported: true
  150. dataDirHostPath: /data/rook
  151. # set the amount of mons to be started
  152. mon:
  153. count: 3
  154. allowMultiplePerNode: true
  155. dashboard:
  156. enabled: true
  157. network:
  158. hostNetwork: false
  159. rbdMirroring:
  160. workers: 0
  161. resources:
  162. storage:
  163. useAllNodes: true
  164. useAllDevices: false
  165. deviceFilter:
  166. location:
  167. config:
  168. databaseSizeMB: "1024"
  169. journalSizeMB: "1024"
  170. osdsPerDevice: "1"

Create the cluster:

  1. kubectl create -f cluster.yaml

Make sure the following pods are Running.

  1. $ kubectl -n rook-ceph get pod
  2. NAME READY STATUS RESTARTS AGE
  3. opa-7458bf7dc6-g72tt 1/1 Running 0 7m
  4. rook-ceph-mgr-a-77c8bc845c-t4j9s 1/1 Running 0 4m
  5. rook-ceph-mon-a-79f5886d9c-mgkpc 1/1 Running 0 5m
  6. rook-ceph-mon-b-68dcffc7cb-xcdts 1/1 Running 0 4m
  7. rook-ceph-mon-c-844f9d4fbd-tz9pn 1/1 Running 0 4m
  8. rook-ceph-osd-0-7479c85878-mbrhd 1/1 Running 0 3m
  9. rook-ceph-osd-prepare-minikube-4mm7c 0/2 Completed 0 4m

5. Configure Ceph to use OPA

The Ceph Object Gateway needs to be configured to use OPA for authorization decisions. The following configuration options are available for the OPA integration with the gateway:

  1. rgw use opa authz = {use opa server to authorize client requests}
  2. rgw opa url = {opa server url:opa server port}
  3. rgw opa token = {opa bearer token}
  4. rgw opa verify ssl = {verify opa server ssl certificate}

More information on the OPA - Ceph Object Gateway integration can be found in the Ceph docs.

When the Rook Operator creates a cluster, a placeholder ConfigMap is created that can be used to override Ceph’s configuration settings.

Update the ConfigMap to include the OPA-related options.

  1. kubectl -n rook-ceph edit configmap rook-config-override

Modify the settings and save.

  1. data:
  2. config: |
  3. [client.radosgw.gateway]
  4. rgw use opa authz = true
  5. rgw opa url = opa.rook-ceph:8181/v1/data/ceph/authz/allow

6. Create the Ceph Object Store

Save the object store spec as object.yaml:

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStore
  3. metadata:
  4. name: my-store
  5. namespace: rook-ceph
  6. spec:
  7. metadataPool:
  8. failureDomain: host
  9. replicated:
  10. size: 1
  11. dataPool:
  12. failureDomain: osd
  13. replicated:
  14. size: 1
  15. gateway:
  16. type: s3
  17. sslCertificateRef:
  18. port: 80
  19. securePort:
  20. instances: 1
  21. allNodes: false
  22. placement:
  23. resources:
  1. kubectl create -f object.yaml

When the object store is created, the RGW service with the S3 API will be started in the cluster. The Rook operator will create all the pools and other resources necessary to start the service.

Check that the RGW pod is Running.

  1. kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

Rook sets up the object storage so pods will have access internal to the cluster. Create a new service for external access. We will need the external RGW service for exercising our OPA policy.

Save the external service as rgw-external.yaml:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. name: rook-ceph-rgw-my-store-external
  5. namespace: rook-ceph
  6. labels:
  7. app: rook-ceph-rgw
  8. rook_cluster: rook-ceph
  9. rook_object_store: my-store
  10. spec:
  11. ports:
  12. - name: rgw
  13. port: 80
  14. protocol: TCP
  15. targetPort: 80
  16. selector:
  17. app: rook-ceph-rgw
  18. rook_cluster: rook-ceph
  19. rook_object_store: my-store
  20. sessionAffinity: None
  21. type: NodePort

Create the external service.

  1. kubectl create -f rgw-external.yaml

Check that both the internal and external RGW services are Running.

  1. kubectl -n rook-ceph get service rook-ceph-rgw-my-store rook-ceph-rgw-my-store-external

7. Create Object Store Users

Create two object store users Alice and Bob.

object-user-alice.yaml

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStoreUser
  3. metadata:
  4. name: alice
  5. namespace: rook-ceph
  6. spec:
  7. store: my-store
  8. displayName: "Alice"

object-user-bob.yaml

  1. apiVersion: ceph.rook.io/v1
  2. kind: CephObjectStoreUser
  3. metadata:
  4. name: bob
  5. namespace: rook-ceph
  6. spec:
  7. store: my-store
  8. displayName: "Bob"

Now create the users.

  1. kubectl create -f object-user-alice.yaml
  2. kubectl create -f object-user-bob.yaml

When the object store user is created the Rook operator will create the RGW user on the object store my-store, and store the user’s Access Key and Secret Key in a Kubernetes secret in the namespace rook-ceph.

8. Understanding the OPA policy

As we saw earlier, the OPA spec contained a ConfigMap that defined the policy to be used to authorize requests received by the Ceph Object Gateway. Below is the policy:

authz.rego

  1. package ceph.authz
  2. default allow = false
  3. #-----------------------------------------------------------------------------
  4. # Data structures containing location info about users and buckets.
  5. # In real-world deployments, these data structures could be loaded into
  6. # OPA as raw JSON data. The JSON data could also be pulled from external
  7. # sources like AD, Git, etc.
  8. #-----------------------------------------------------------------------------
  9. # user-location information
  10. user_location = {
  11. "alice": "UK",
  12. "bob": "USA"
  13. }
  14. # bucket-location information
  15. bucket_location = {
  16. "supersecretbucket": "USA"
  17. }
  18. # Allow access to bucket in same location as user.
  19. allow {
  20. input.method == "HEAD"
  21. is_user_in_bucket_location(input.user_info.user_id, input.bucket_info.bucket.name)
  22. }
  23. allow {
  24. input.method == "GET"
  25. }
  26. allow {
  27. input.method == "PUT"
  28. input.user_info.display_name == "Bob"
  29. }
  30. allow {
  31. input.method == "DELETE"
  32. input.user_info.display_name == "Bob"
  33. }
  34. # Check if the user and the bucket being accessed belong to the same
  35. # location.
  36. is_user_in_bucket_location(user, bucket) {
  37. user_location[user] == bucket_location[bucket]
  38. }

The above policy will restrict a user from accessing a bucket whose location does not match the user’s location.. The user’s and bucket’s location is hardcoded in the policy for simplicity and in the real-world can be fetched from external sources or pushed into OPA using it’s REST API.

In the above policy, Bob's location is USA while Alice's is UK. Since the bucket supersecretbucket is located in the USA, Alice should not be able to access it.

9. Create the S3 access test script

The below Python S3 access test script connects to the Ceph Object Store Gateway to perform actions such as creating and deleting buckets.

You will need to install the python-boto package to run the test script.

Save the test script as s3test.py:

  1. #!/usr/bin/env python
  2. import sys
  3. import boto.s3.connection
  4. from boto.s3.key import Key
  5. import os
  6. def create_bucket(conn, bucket_name):
  7. try:
  8. bucket = conn.create_bucket(bucket_name)
  9. except Exception as e:
  10. print 'Unable to create bucket: Forbidden'
  11. def delete_bucket(conn, bucket_name):
  12. try:
  13. bucket = conn.delete_bucket(bucket_name)
  14. except Exception as e:
  15. print 'Unable to delete bucket: Forbidden'
  16. def list_bucket(conn):
  17. buckets = conn.get_all_buckets()
  18. if len(buckets) == 0:
  19. print 'No Buckets'
  20. return
  21. for bucket in buckets:
  22. print "{name} {created}".format(
  23. name=bucket.name,
  24. created=bucket.creation_date,
  25. )
  26. def upload_data(conn, bucket_name, data):
  27. bucket = conn.get_bucket(bucket_name)
  28. k = Key(bucket)
  29. k.key = 'foobar'
  30. k.set_contents_from_string(data)
  31. def download_data(conn, user, bucket_name):
  32. try:
  33. bucket = conn.get_bucket(bucket_name)
  34. except Exception as e:
  35. print 'Not allowed to access bucket "{}": User "{}" not in the same location as bucket "{}"'.format(bucket_name, user, bucket_name)
  36. else:
  37. k = Key(bucket)
  38. k.key = 'foobar'
  39. print k.get_contents_as_string()
  40. if __name__ == '__main__':
  41. user = sys.argv[1]
  42. action = sys.argv[2]
  43. if len(sys.argv) == 4:
  44. bucket_name = sys.argv[3]
  45. if len(sys.argv) == 5:
  46. bucket_name = sys.argv[3]
  47. data = sys.argv[4]
  48. access_key_env_name = '{}_ACCESS_KEY'.format(user.upper())
  49. secret_key_env_name = '{}_SECRET_KEY'.format(user.upper())
  50. access_key = os.getenv(access_key_env_name)
  51. secret_key = os.getenv(secret_key_env_name)
  52. conn = boto.connect_s3(
  53. aws_access_key_id=access_key,
  54. aws_secret_access_key=secret_key,
  55. host=os.getenv("HOST"), port=int(os.getenv("PORT")),
  56. is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
  57. )
  58. if action == 'create':
  59. create_bucket(conn, bucket_name)
  60. if action == 'list':
  61. list_bucket(conn)
  62. if action == 'delete':
  63. delete_bucket(conn, bucket_name)
  64. if action == 'upload_data':
  65. upload_data(conn, bucket_name, data)
  66. if action == 'download_data':
  67. download_data(conn, user, bucket_name)

The script needs the following environment variables:

  • HOST - Hostname of the machine running the RGW service in the cluster.
  • PORT - Port of the RGW service.
  • <USER>_ACCESS_KEY - USER’s ACCESS_KEY
  • <USER>_SECRET_KEY - USER’s SECRET_KEY

We previously created a service to provide external access to the RGW.

  1. $ kubectl -n rook-ceph get service rook-ceph-rgw-my-store-external
  2. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  3. rook-ceph-rgw-my-store-external NodePort 10.111.198.148 <none> 80:31765/TCP 2h

Internally the RGW service is running on port 80. The external port in this case is 31765.

Set the HOST and PORT environment variables:

  1. export HOST=$(minikube ip)
  2. export PORT=$(kubectl -n rook-ceph get service rook-ceph-rgw-my-store-external -o jsonpath='{.spec.ports[?(@.name=="rgw")].nodePort}')

Get Alice’s and Bob’s ACCESS_KEY and SECRET_KEY from the Kubernetes Secret and set the following environment variables:

  1. export ALICE_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-my-store-alice -n rook-ceph -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode)
  2. export ALICE_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-my-store-alice -n rook-ceph -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode)
  3. export BOB_ACCESS_KEY=$(kubectl get secret rook-ceph-object-user-my-store-bob -n rook-ceph -o yaml | grep AccessKey | awk '{print $2}' | base64 --decode)
  4. export BOB_SECRET_KEY=$(kubectl get secret rook-ceph-object-user-my-store-bob -n rook-ceph -o yaml | grep SecretKey | awk '{print $2}' | base64 --decode)

Now let’s create a bucket and add some data to it.

  • First, Bob creates a bucket supersecretbucket

    1. python s3test.py Bob create supersecretbucket
    • List the bucket just created
    1. python s3test.py Bob list

    The output will be something like:

    1. supersecretbucket 2019-01-14T21:18:03.872Z
    • Add some data to the bucket supersecretbucket
    1. python s3test.py Bob upload_data supersecretbucket "This is some secret data"

10. Exercise the OPA policy

To recap, the policy we are going to test will restrict a user from accessing a bucket whose location does not match the user’s location..

Check that Alice cannot access the contents of the bucket supersecretbucket.

  1. python s3test.py Alice download_data supersecretbucket

Since Alice is located in UK and and the bucket supersecretbucket in the USA, she would be denied access.

Check that Bob can access the contents of the bucket supersecretbucket.

  1. python s3test.py Bob download_data supersecretbucket

Since Bob and the bucket supersecretbucket are both located in the USA, Bob is granted access to the contents in the bucket.

Wrap Up

Congratulations for finishing the tutorial!

This tutorial showed how OPA can be used to enforce custom policies over the S3 API to the Ceph Storage Cluster. You can modify OPA’s polices to get greater control over the actions performed on the Ceph Object Storage without making any changes to Ceph.

This tutorial also showed how OPA can seamlessly work with Rook without any modifications to Rook’s components.