OpenEBS for NuoDB

OpenEBS and Nuodb

Introduction

NuoDB’s distributed SQL database combines the elastic scale and continuous availability of the cloud with the transactional consistency and durability that databases of record demand. NuoDB is deployed usually as a StatefulSet on Kubernetes and requires persistent storage for each instance of NuoDB StorageManager instance. OpenEBS provides persistent volumes on the fly when StorageManagers are scaled up.

Advantages of using OpenEBS for NuoDB database:

  • No need to manage the local disks, they are managed by OpenEBS
  • Large size PVs can be provisioned by OpenEBS and NuoDB
  • Start with small storage and add disks as needed on the fly. Sometimes NuoDB instances are scaled up because of capacity on the nodes. With OpenEBS persistent volumes, capacity can be thin provisioned and disks can be added to OpenEBS on the fly without disruption of service
  • If required, take backup of the NuoDB data periodically and back them up to S3 or any object storage so that restoration of the same data is possible to the same or any other Kubernetes cluster

Note: NuoDB can be deployed both as deployment or as statefulset. When NuoDB deployed as statefulset, you don’t need to replicate the data again at OpenEBS level. When NuoDB is deployed as deployment, consider 3 OpenEBS replicas, choose the StorageClass accordingly.

Deployment model

OpenEBS and NuoDB

Configuration workflow

  1. Install OpenEBS

    If OpenEBS is not installed in your K8s cluster, this can done from here. If OpenEBS is already installed, go to the next step.

  2. Configure cStor Pool

    After OpenEBS installation, cStor pool has to be configured. If cStor Pool is not configured in your OpenEBS cluster, this can be done from here. Sample YAML named openebs-config.yaml for configuring cStor Pool is provided in the Configuration details below. During cStor Pool creation, make sure that the maxPools parameter is set to >=3. If cStor pool is already configured, go to the next step.

  3. Create Storage Class

    You must configure a StorageClass to provision cStor volume on given cStor pool. StorageClass is the interface through which most of the OpenEBS storage policies are defined. In this solution we are using a StorageClass to consume the cStor Pool which is created using external disks attached on the Nodes. Since NuoDB is a StatefulSet application, it requires only single storage replica. So cStor volume replicaCount is =1. Sample YAML named openebs-sc-disk.yaml to consume cStor pool with cStor volume replica count as 1 is provided in the configuration details below.

  4. Download the YAML spec files from OpenEBS litmus repository.

    1. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb.yaml
    2. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb-sm.yaml
    3. wget https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/nuodb-te.yaml
  5. Edit the above YAML spec files and update them with correct StorageClass. For example, replace openebs-cstor-sparse with openebs-cstor-disk

  6. Launch and test NuoDB

    Create a namespace called testns and apply following YAML files to deploy NuoDB application. Sample YAML files are provided in the Configuration details below.

    1. kubectl apply -f nuodb.yaml -n testns
    2. kubectl apply -f nuodb-sm.yaml -n testns
    3. kubectl apply -f nuodb-te.yaml -n testns

Reference at openebs.ci

Deployment YAML spec files for NuoDB and OpenEBS resources are found here

OpenEBS-CI dashboard of NuoDB

Live access to NuoDB dashboard

Post deployment Operations

Monitor OpenEBS Volume size

It is not seamless to increase the cStor volume size (refer to the roadmap item). Hence, it is recommended that sufficient size is allocated during the initial configuration.

Monitor cStor Pool size

As in most cases, cStor pool may not be dedicated to just NuoDB database alone. It is recommended to watch the pool capacity and add more disks to the pool before it hits 80% threshold. See cStorPool metrics

Configuration details

openebs-config.yaml

  1. #Use the following YAMLs to create a cStor Storage Pool.
  2. # and associated storage class.
  3. apiVersion: openebs.io/v1alpha1
  4. kind: StoragePoolClaim
  5. metadata:
  6. name: cstor-disk
  7. spec:
  8. name: cstor-disk
  9. type: disk
  10. poolSpec:
  11. poolType: striped
  12. # NOTE - Appropriate disks need to be fetched using `kubectl get blockdevices -n openebs`
  13. # `Block devices` is a custom resource supported by OpenEBS with `node-disk-manager`
  14. # as the disk operator
  15. # Replace the following with actual disk CRs from your cluster `kubectl get blockdevices -n openebs`
  16. # Uncomment the below lines after updating the actual disk names.
  17. blockDevices:
  18. blockDeviceList:
  19. # Replace the following with actual disk CRs from your cluster from `kubectl get blockdevices -n openebs`
  20. # - blockdevice-69cdfd958dcce3025ed1ff02b936d9b4
  21. # - blockdevice-891ad1b581591ae6b54a36b5526550a2
  22. # - blockdevice-ceaab442d802ca6aae20c36d20859a0b
  23. ---

openebs-sc-disk.yaml

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4. name: openebs-cstor-disk
  5. annotations:
  6. openebs.io/cas-type: cstor
  7. cas.openebs.io/config: |
  8. - name: StoragePoolClaim
  9. value: "cstor-disk"
  10. - name: ReplicaCount
  11. value: "1"
  12. provisioner: openebs.io/provisioner-iscsi
  13. reclaimPolicy: Delete
  14. ---

nuodb.yaml

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. annotations:
  6. description: |
  7. Headless service permitting us to retrieve DNS A records that point directly to the pods backing the service.
  8. "service.alpha.kubernetes.io/tolerate-unready-endpoints": "true"
  9. name: domain
  10. labels:
  11. app: nuodb
  12. group: nuodb
  13. spec:
  14. clusterIP: None
  15. ports:
  16. - { name: 8888-tcp, port: 8888, protocol: TCP, targetPort: 8888 }
  17. - { name: 48004-tcp, port: 48004, protocol: TCP, targetPort: 48004 }
  18. - { name: 48005-tcp, port: 48005, protocol: TCP, targetPort: 48005 }
  19. selector:
  20. dns: domain
  21. sessionAffinity: None
  22. type: ClusterIP
  23. status:
  24. loadBalancer: {}
  25. ---
  26. apiVersion: v1
  27. kind: Service
  28. metadata:
  29. annotations:
  30. description: "Service (and load-balancer) for Admin pods."
  31. labels:
  32. app: nuodb
  33. group: nuodb
  34. name: admin
  35. spec:
  36. ports:
  37. - { name: 8888-tcp, port: 8888, protocol: TCP, targetPort: 8888 }
  38. - { name: 48004-tcp, port: 48004, protocol: TCP, targetPort: 48004 }
  39. - { name: 48005-tcp, port: 48005, protocol: TCP, targetPort: 48005 }
  40. selector:
  41. app: admin
  42. sessionAffinity: None
  43. type: LoadBalancer
  44. status:
  45. loadBalancer: {}
  46. ---
  47. apiVersion: apps/v1beta1
  48. kind: StatefulSet
  49. metadata:
  50. name: admin
  51. labels:
  52. app: nuodb
  53. group: nuodb
  54. spec:
  55. serviceName: domain
  56. replicas: 1
  57. selector:
  58. matchLabels:
  59. affinity: admin
  60. template:
  61. metadata:
  62. labels:
  63. dns: domain
  64. app: admin
  65. affinity: admin
  66. spec:
  67. securityContext:
  68. runAsUser: 0
  69. fsGroup: 0
  70. containers:
  71. - name: admin
  72. image: nuodb/nuodb-ce:latest
  73. imagePullPolicy: IfNotPresent
  74. ports:
  75. - { containerPort: 8888, protocol: TCP }
  76. - { containerPort: 48004, protocol: TCP }
  77. - { containerPort: 48005, protocol: TCP }
  78. resources:
  79. requests:
  80. memory: "512Mi"
  81. cpu: "0.5"
  82. limits:
  83. memory: "1024Mi"
  84. cpu: "1"
  85. env:
  86. - name: NODE_NAME
  87. valueFrom:
  88. fieldRef:
  89. fieldPath: spec.nodeName
  90. - name: POD_NAME
  91. valueFrom:
  92. fieldRef:
  93. fieldPath: metadata.name
  94. - name: NAMESPACE
  95. valueFrom:
  96. fieldRef:
  97. fieldPath: metadata.namespace
  98. - { name: NUODB_DOMAIN_ENTRYPOINT, value: "admin-0.domain" }
  99. - { name: NUODB_ALT_ADDRESS, value: "$(POD_NAME).domain.testns.svc" }
  100. - { name: NUODB_VARDIR, value: "/var/opt/nuodb/$(POD_NAME).testns" }
  101. args: [ "nuoadmin" ]
  102. livenessProbe:
  103. initialDelaySeconds: 30
  104. periodSeconds: 15
  105. tcpSocket:
  106. port: 8888
  107. readinessProbe:
  108. initialDelaySeconds: 30
  109. periodSeconds: 15
  110. exec:
  111. command: [ "nuodocker", "check", "servers" ]
  112. volumeMounts:
  113. - name: raftlog
  114. mountPath: /var/opt/nuodb
  115. - name: log-volume
  116. mountPath: /var/log/nuodb
  117. volumes:
  118. - name: log-volume
  119. emptyDir: {}
  120. volumeClaimTemplates:
  121. - metadata:
  122. name: raftlog
  123. annotations:
  124. volume.beta.kubernetes.io/storage-class: openebs-cstor-sparse
  125. spec:
  126. accessModes: [ "ReadWriteOnce" ]
  127. resources:
  128. requests:
  129. storage: 5G
  130. ---
  131. apiVersion: v1
  132. kind: Service
  133. metadata:
  134. annotations:
  135. description: "Service for redirect."
  136. labels:
  137. app: insights
  138. group: nuodb
  139. name: insights-server
  140. spec:
  141. ports:
  142. - { name: 8080-tcp, port: 8080, protocol: TCP, targetPort: 8080 }
  143. selector:
  144. app: insights
  145. group: nuodb
  146. sessionAffinity: None
  147. type: LoadBalancer
  148. status:
  149. loadBalancer: {}
  150. ---
  151. apiVersion: v1
  152. kind: ConfigMap
  153. metadata:
  154. annotations:
  155. description: "Config map for metrics input and insights output"
  156. name: insights-configmap
  157. labels:
  158. group: nuodb
  159. data:
  160. nuoca.yml.template: |
  161. ---
  162. NUOCA_LOGFILE: /var/log/nuodb/nuoca.log
  163. INPUT_PLUGINS:
  164. - NuoAdminNuoMon:
  165. description : Collection from NuoDB engines
  166. nuocaCollectionName: NuoMon
  167. api_server: https://domain:8888
  168. client_key: /etc/nuodb/keys/nuocmd.pem
  169. OUTPUT_PLUGINS:
  170. - RestClient:
  171. url: ${INSIGHTS_INGEST_URL}
  172. nuoinsights: |
  173. #!/bin/bash
  174. . ${NUODB_HOME}/etc/nuodb_setup.sh
  175. : ${NUOCMD_API_SERVER:=https://domain:8888}
  176. SCRIPT=$(cat <<EOF
  177. from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
  178. from nuoca import NuoCA as nuoca
  179. import nuodb_mgmt
  180. PORT_NUMBER = 8080
  181. def get_insights_dashboard():
  182. conn = nuodb_mgmt.AdminConnection('${NUOCMD_API_SERVER}', '/etc/nuodb/keys/nuocmd.pem')
  183. metadata = nuoca.get_insights_metadata(conn)
  184. return metadata.get(nuoca.INSIGHTS_DASHBOARD_KEY,None)
  185. #ingest_url = metadata[nuoca.INSIGHTS_URL_KEY]
  186. #subscriber_id = metadata[nuoca.INSIGHTS_ID_KEY]
  187. #return ingest_url[:-6] + subscriber_id
  188. class myHandler(BaseHTTPRequestHandler):
  189. def do_GET(self):
  190. new_path = get_insights_dashboard()
  191. if new_path and new_path != "None":
  192. self.send_response(307)
  193. self.send_header('Location', new_path)
  194. self.end_headers()
  195. else:
  196. self.send_response(404)
  197. self.end_headers()
  198. return
  199. try:
  200. server = HTTPServer(('', PORT_NUMBER), myHandler)
  201. print 'Started httpserver on port ' , PORT_NUMBER
  202. #Wait forever for incoming http requests
  203. server.serve_forever()
  204. except KeyboardInterrupt:
  205. print '^C received, shutting down the web server'
  206. server.socket.close()
  207. EOF
  208. )
  209. export PYTHONPATH=/opt/nuodb/drivers/pynuoadmin
  210. exec /opt/nuodb/etc/python/nuopython -c "${SCRIPT}"
  211. ---
  212. apiVersion: v1
  213. kind: Pod
  214. metadata:
  215. name: nuodb-insights
  216. labels:
  217. app: insights
  218. group: nuodb
  219. spec:
  220. initContainers:
  221. - name: optin
  222. image: nuodb/nuodb-ce:latest
  223. imagePullPolicy: IfNotPresent
  224. args:
  225. - "/bin/sh"
  226. - "-c"
  227. - "[ \"${OPT_IN}\" == \"true\" ] && nuoca enable insights --connect-timeout 300 || nuoca disable insights --connect-timeout 300"
  228. env:
  229. - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  230. containers:
  231. - name: insights
  232. image: nuodb/nuodb-ce:latest
  233. imagePullPolicy: IfNotPresent
  234. args: [ "nuoca", "start" , "nuoca", "--insights" ]
  235. env:
  236. - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  237. volumeMounts:
  238. - name: log-volume
  239. mountPath: /var/log/nuodb
  240. - name: config-insights
  241. mountPath: /etc/nuodb/nuoca.yml.template
  242. subPath: nuoca.yml.template
  243. - name: insights-server
  244. image: nuodb/nuodb-ce:latest
  245. imagePullPolicy: IfNotPresent
  246. args: [ "nuoinsights" ]
  247. ports:
  248. - { name: 8080-tcp, containerPort: 8080, protocol: TCP }
  249. env:
  250. - { name: NUOCMD_API_SERVER , value: "https://domain:8888" }
  251. volumeMounts:
  252. - name: log-volume
  253. mountPath: /var/log/nuodb
  254. - name: nuoinsights
  255. mountPath: /usr/local/bin/nuoinsights
  256. subPath: nuoinsights
  257. volumes:
  258. - name: log-volume
  259. emptyDir: {}
  260. - name: config-insights
  261. configMap:
  262. name: insights-configmap
  263. - name: nuoinsights
  264. configMap:
  265. name: insights-configmap
  266. defaultMode: 0754

nuodb-sm.yaml

  1. ---
  2. apiVersion: apps/v1beta1
  3. kind: StatefulSet
  4. metadata:
  5. name: sm
  6. labels:
  7. app: nuodb
  8. group: nuodb
  9. spec:
  10. serviceName: sm
  11. replicas: 1
  12. selector:
  13. matchLabels:
  14. nodetype: sm
  15. database: nuodb
  16. app: sm
  17. group: nuodb
  18. podManagementPolicy: Parallel
  19. template:
  20. metadata:
  21. labels:
  22. nodetype: sm
  23. database: nuodb
  24. app: sm
  25. group: nuodb
  26. spec:
  27. securityContext:
  28. runAsUser: 0
  29. fsGroup: 0
  30. containers:
  31. - name: sm
  32. image: nuodb/nuodb-ce:latest
  33. imagePullPolicy: IfNotPresent
  34. args: [ "nuosm", "--servers-ready-timeout", "300" ]
  35. env:
  36. - name: NODE_NAME
  37. valueFrom:
  38. fieldRef:
  39. fieldPath: spec.nodeName
  40. - name: POD_NAME
  41. valueFrom:
  42. fieldRef:
  43. fieldPath: metadata.name
  44. - name: NAMESPACE
  45. valueFrom:
  46. fieldRef:
  47. fieldPath: metadata.namespace
  48. - { name: DB_NAME, value: "nuodb"}
  49. - { name: DB_USER, value: "dba"}
  50. - { name: DB_PASSWORD, value: "goalie"}
  51. - { name: NUOCMD_API_SERVER, value: "admin.testns.svc:8888"}
  52. - { name: PEER_ADDRESS, value: "admin.testns.svc" }
  53. - { name: NUODB_OPTIONS, value: "mem 1g" }
  54. ports:
  55. - containerPort: 48006
  56. protocol: TCP
  57. resources:
  58. requests:
  59. memory: "1Gi"
  60. cpu: "200m"
  61. terminationMessagePath: /dev/termination-log
  62. terminationMessagePolicy: File
  63. volumeMounts:
  64. - mountPath: /var/opt/nuodb/archive
  65. name: archive
  66. volumeClaimTemplates:
  67. - metadata:
  68. name: archive
  69. annotations:
  70. volume.beta.kubernetes.io/storage-class: openebs-cstor-sparse
  71. spec:
  72. accessModes: [ "ReadWriteOnce" ]
  73. resources:
  74. requests:
  75. storage: 5G

nuodb-te.yaml

  1. ---
  2. apiVersion: apps/v1beta1
  3. kind: Deployment
  4. metadata:
  5. name: te
  6. labels:
  7. app: nuodb
  8. group: nuodb
  9. deploymentconfig: te
  10. spec:
  11. replicas: 1
  12. template:
  13. metadata:
  14. labels:
  15. app: te
  16. deploymentconfig: te
  17. database: nuodb
  18. nodetype: te
  19. spec:
  20. containers:
  21. -
  22. name: te
  23. image: nuodb/nuodb-ce:latest
  24. imagePullPolicy: IfNotPresent
  25. env:
  26. - name: NODE_NAME
  27. valueFrom:
  28. fieldRef:
  29. fieldPath: spec.nodeName
  30. - name: POD_NAME
  31. valueFrom:
  32. fieldRef:
  33. fieldPath: metadata.name
  34. - name: NAMESPACE
  35. valueFrom:
  36. fieldRef:
  37. fieldPath: metadata.namespace
  38. - { name: DB_NAME, value: "nuodb" }
  39. - { name: NUOCMD_API_SERVER, value: "admin.testns.svc:8888"}
  40. - { name: PEER_ADDRESS, value: "admin.testns.svc" }
  41. - { name: NUODB_LABELS, value: "" }
  42. - { name: NUODB_OPTIONS, value: "mem 1g" }
  43. args: [ "nuote", "--servers-ready-timeout", "300", "--database-created-timeout", "300" ]
  44. ports:
  45. - containerPort: 48006
  46. protocol: TCP
  47. resources:
  48. requests:
  49. memory: "1Gi"
  50. cpu: "200m"
  51. terminationMessagePath: /dev/termination-log
  52. terminationMessagePolicy: File
  53. volumeMounts:
  54. - name: logdir
  55. mountPath: /var/log/nuodb
  56. volumes:
  57. - name: logdir
  58. emptyDir: {}

See Also:

OpenEBS architecture

OpenEBS use cases

cStor pools overview