Run a Replicated Stateful Application

This page shows how to run a replicated stateful application using a StatefulSet. This application is a replicated MySQL database. The example topology has a single primary server and multiple replicas, using asynchronous row-based replication.

Note: This is not a production configuration. MySQL settings remain on insecure defaults to keep the focus on general patterns for running stateful applications in Kubernetes.

Before you begin

Objectives

  • Deploy a replicated MySQL topology with a StatefulSet.
  • Send MySQL client traffic.
  • Observe resistance to downtime.
  • Scale the StatefulSet up and down.

Deploy MySQL

The example MySQL deployment consists of a ConfigMap, two Services, and a StatefulSet.

Create a ConfigMap

Create the ConfigMap from the following YAML configuration file:

application/mysql/mysql-configmap.yaml Run a Replicated Stateful Application - 图1

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: mysql
  5. labels:
  6. app: mysql
  7. app.kubernetes.io/name: mysql
  8. data:
  9. primary.cnf: |
  10. # Apply this config only on the primary.
  11. [mysqld]
  12. log-bin
  13. replica.cnf: |
  14. # Apply this config only on replicas.
  15. [mysqld]
  16. super-read-only
  1. kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml

This ConfigMap provides my.cnf overrides that let you independently control configuration on the primary MySQL server and its replicas. In this case, you want the primary server to be able to serve replication logs to replicas and you want replicas to reject any writes that don’t come via replication.

There’s nothing special about the ConfigMap itself that causes different portions to apply to different Pods. Each Pod decides which portion to look at as it’s initializing, based on information provided by the StatefulSet controller.

Create Services

Create the Services from the following YAML configuration file:

application/mysql/mysql-services.yaml Run a Replicated Stateful Application - 图2

  1. # Headless service for stable DNS entries of StatefulSet members.
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. name: mysql
  6. labels:
  7. app: mysql
  8. app.kubernetes.io/name: mysql
  9. spec:
  10. ports:
  11. - name: mysql
  12. port: 3306
  13. clusterIP: None
  14. selector:
  15. app: mysql
  16. ---
  17. # Client service for connecting to any MySQL instance for reads.
  18. # For writes, you must instead connect to the primary: mysql-0.mysql.
  19. apiVersion: v1
  20. kind: Service
  21. metadata:
  22. name: mysql-read
  23. labels:
  24. app: mysql
  25. app.kubernetes.io/name: mysql
  26. readonly: "true"
  27. spec:
  28. ports:
  29. - name: mysql
  30. port: 3306
  31. selector:
  32. app: mysql
  1. kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml

The headless Service provides a home for the DNS entries that the StatefulSet controllers creates for each Pod that’s part of the set. Because the headless Service is named mysql, the Pods are accessible by resolving <pod-name>.mysql from within any other Pod in the same Kubernetes cluster and namespace.

The client Service, called mysql-read, is a normal Service with its own cluster IP that distributes connections across all MySQL Pods that report being Ready. The set of potential endpoints includes the primary MySQL server and all replicas.

Note that only read queries can use the load-balanced client Service. Because there is only one primary MySQL server, clients should connect directly to the primary MySQL Pod (through its DNS entry within the headless Service) to execute writes.

Create the StatefulSet

Finally, create the StatefulSet from the following YAML configuration file:

application/mysql/mysql-statefulset.yaml Run a Replicated Stateful Application - 图3

  1. apiVersion: apps/v1
  2. kind: StatefulSet
  3. metadata:
  4. name: mysql
  5. spec:
  6. selector:
  7. matchLabels:
  8. app: mysql
  9. app.kubernetes.io/name: mysql
  10. serviceName: mysql
  11. replicas: 3
  12. template:
  13. metadata:
  14. labels:
  15. app: mysql
  16. app.kubernetes.io/name: mysql
  17. spec:
  18. initContainers:
  19. - name: init-mysql
  20. image: mysql:5.7
  21. command:
  22. - bash
  23. - "-c"
  24. - |
  25. set -ex
  26. # Generate mysql server-id from pod ordinal index.
  27. [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
  28. ordinal=${BASH_REMATCH[1]}
  29. echo [mysqld] > /mnt/conf.d/server-id.cnf
  30. # Add an offset to avoid reserved server-id=0 value.
  31. echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
  32. # Copy appropriate conf.d files from config-map to emptyDir.
  33. if [[ $ordinal -eq 0 ]]; then
  34. cp /mnt/config-map/primary.cnf /mnt/conf.d/
  35. else
  36. cp /mnt/config-map/replica.cnf /mnt/conf.d/
  37. fi
  38. volumeMounts:
  39. - name: conf
  40. mountPath: /mnt/conf.d
  41. - name: config-map
  42. mountPath: /mnt/config-map
  43. - name: clone-mysql
  44. image: gcr.io/google-samples/xtrabackup:1.0
  45. command:
  46. - bash
  47. - "-c"
  48. - |
  49. set -ex
  50. # Skip the clone if data already exists.
  51. [[ -d /var/lib/mysql/mysql ]] && exit 0
  52. # Skip the clone on primary (ordinal index 0).
  53. [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
  54. ordinal=${BASH_REMATCH[1]}
  55. [[ $ordinal -eq 0 ]] && exit 0
  56. # Clone data from previous peer.
  57. ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
  58. # Prepare the backup.
  59. xtrabackup --prepare --target-dir=/var/lib/mysql
  60. volumeMounts:
  61. - name: data
  62. mountPath: /var/lib/mysql
  63. subPath: mysql
  64. - name: conf
  65. mountPath: /etc/mysql/conf.d
  66. containers:
  67. - name: mysql
  68. image: mysql:5.7
  69. env:
  70. - name: MYSQL_ALLOW_EMPTY_PASSWORD
  71. value: "1"
  72. ports:
  73. - name: mysql
  74. containerPort: 3306
  75. volumeMounts:
  76. - name: data
  77. mountPath: /var/lib/mysql
  78. subPath: mysql
  79. - name: conf
  80. mountPath: /etc/mysql/conf.d
  81. resources:
  82. requests:
  83. cpu: 500m
  84. memory: 1Gi
  85. livenessProbe:
  86. exec:
  87. command: ["mysqladmin", "ping"]
  88. initialDelaySeconds: 30
  89. periodSeconds: 10
  90. timeoutSeconds: 5
  91. readinessProbe:
  92. exec:
  93. # Check we can execute queries over TCP (skip-networking is off).
  94. command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
  95. initialDelaySeconds: 5
  96. periodSeconds: 2
  97. timeoutSeconds: 1
  98. - name: xtrabackup
  99. image: gcr.io/google-samples/xtrabackup:1.0
  100. ports:
  101. - name: xtrabackup
  102. containerPort: 3307
  103. command:
  104. - bash
  105. - "-c"
  106. - |
  107. set -ex
  108. cd /var/lib/mysql
  109. # Determine binlog position of cloned data, if any.
  110. if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
  111. # XtraBackup already generated a partial "CHANGE MASTER TO" query
  112. # because we're cloning from an existing replica. (Need to remove the tailing semicolon!)
  113. cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
  114. # Ignore xtrabackup_binlog_info in this case (it's useless).
  115. rm -f xtrabackup_slave_info xtrabackup_binlog_info
  116. elif [[ -f xtrabackup_binlog_info ]]; then
  117. # We're cloning directly from primary. Parse binlog position.
  118. [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
  119. rm -f xtrabackup_binlog_info xtrabackup_slave_info
  120. echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
  121. MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
  122. fi
  123. # Check if we need to complete a clone by starting replication.
  124. if [[ -f change_master_to.sql.in ]]; then
  125. echo "Waiting for mysqld to be ready (accepting connections)"
  126. until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
  127. echo "Initializing replication from clone position"
  128. mysql -h 127.0.0.1 \
  129. -e "$(<change_master_to.sql.in), \
  130. MASTER_HOST='mysql-0.mysql', \
  131. MASTER_USER='root', \
  132. MASTER_PASSWORD='', \
  133. MASTER_CONNECT_RETRY=10; \
  134. START SLAVE;" || exit 1
  135. # In case of container restart, attempt this at-most-once.
  136. mv change_master_to.sql.in change_master_to.sql.orig
  137. fi
  138. # Start a server to send backups when requested by peers.
  139. exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
  140. "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
  141. volumeMounts:
  142. - name: data
  143. mountPath: /var/lib/mysql
  144. subPath: mysql
  145. - name: conf
  146. mountPath: /etc/mysql/conf.d
  147. resources:
  148. requests:
  149. cpu: 100m
  150. memory: 100Mi
  151. volumes:
  152. - name: conf
  153. emptyDir: {}
  154. - name: config-map
  155. configMap:
  156. name: mysql
  157. volumeClaimTemplates:
  158. - metadata:
  159. name: data
  160. spec:
  161. accessModes: ["ReadWriteOnce"]
  162. resources:
  163. requests:
  164. storage: 10Gi
  1. kubectl apply -f https://k8s.io/examples/application/mysql/mysql-statefulset.yaml

You can watch the startup progress by running:

  1. kubectl get pods -l app=mysql --watch

After a while, you should see all 3 Pods become Running:

  1. NAME READY STATUS RESTARTS AGE
  2. mysql-0 2/2 Running 0 2m
  3. mysql-1 2/2 Running 0 1m
  4. mysql-2 2/2 Running 0 1m

Press Ctrl+C to cancel the watch.

Note: If you don’t see any progress, make sure you have a dynamic PersistentVolume provisioner enabled, as mentioned in the prerequisites.

This manifest uses a variety of techniques for managing stateful Pods as part of a StatefulSet. The next section highlights some of these techniques to explain what happens as the StatefulSet creates Pods.

Understanding stateful Pod initialization

The StatefulSet controller starts Pods one at a time, in order by their ordinal index. It waits until each Pod reports being Ready before starting the next one.

In addition, the controller assigns each Pod a unique, stable name of the form <statefulset-name>-<ordinal-index>, which results in Pods named mysql-0, mysql-1, and mysql-2.

The Pod template in the above StatefulSet manifest takes advantage of these properties to perform orderly startup of MySQL replication.

Generating configuration

Before starting any of the containers in the Pod spec, the Pod first runs any init containers in the order defined.

The first init container, named init-mysql, generates special MySQL config files based on the ordinal index.

The script determines its own ordinal index by extracting it from the end of the Pod name, which is returned by the hostname command. Then it saves the ordinal (with a numeric offset to avoid reserved values) into a file called server-id.cnf in the MySQL conf.d directory. This translates the unique, stable identity provided by the StatefulSet into the domain of MySQL server IDs, which require the same properties.

The script in the init-mysql container also applies either primary.cnf or replica.cnf from the ConfigMap by copying the contents into conf.d. Because the example topology consists of a single primary MySQL server and any number of replicas, the script assigns ordinal 0 to be the primary server, and everyone else to be replicas. Combined with the StatefulSet controller’s deployment order guarantee, this ensures the primary MySQL server is Ready before creating replicas, so they can begin replicating.

Cloning existing data

In general, when a new Pod joins the set as a replica, it must assume the primary MySQL server might already have data on it. It also must assume that the replication logs might not go all the way back to the beginning of time. These conservative assumptions are the key to allow a running StatefulSet to scale up and down over time, rather than being fixed at its initial size.

The second init container, named clone-mysql, performs a clone operation on a replica Pod the first time it starts up on an empty PersistentVolume. That means it copies all existing data from another running Pod, so its local state is consistent enough to begin replicating from the primary server.

MySQL itself does not provide a mechanism to do this, so the example uses a popular open-source tool called Percona XtraBackup. During the clone, the source MySQL server might suffer reduced performance. To minimize impact on the primary MySQL server, the script instructs each Pod to clone from the Pod whose ordinal index is one lower. This works because the StatefulSet controller always ensures Pod N is Ready before starting Pod N+1.

Starting replication

After the init containers complete successfully, the regular containers run. The MySQL Pods consist of a mysql container that runs the actual mysqld server, and an xtrabackup container that acts as a sidecar.

The xtrabackup sidecar looks at the cloned data files and determines if it’s necessary to initialize MySQL replication on the replica. If so, it waits for mysqld to be ready and then executes the CHANGE MASTER TO and START SLAVE commands with replication parameters extracted from the XtraBackup clone files.

Once a replica begins replication, it remembers its primary MySQL server and reconnects automatically if the server restarts or the connection dies. Also, because replicas look for the primary server at its stable DNS name (mysql-0.mysql), they automatically find the primary server even if it gets a new Pod IP due to being rescheduled.

Lastly, after starting replication, the xtrabackup container listens for connections from other Pods requesting a data clone. This server remains up indefinitely in case the StatefulSet scales up, or in case the next Pod loses its PersistentVolumeClaim and needs to redo the clone.

Sending client traffic

You can send test queries to the primary MySQL server (hostname mysql-0.mysql) by running a temporary container with the mysql:5.7 image and running the mysql client binary.

  1. kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\
  2. mysql -h mysql-0.mysql <<EOF
  3. CREATE DATABASE test;
  4. CREATE TABLE test.messages (message VARCHAR(250));
  5. INSERT INTO test.messages VALUES ('hello');
  6. EOF

Use the hostname mysql-read to send test queries to any server that reports being Ready:

  1. kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\
  2. mysql -h mysql-read -e "SELECT * FROM test.messages"

You should get output like this:

  1. Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false
  2. +---------+
  3. | message |
  4. +---------+
  5. | hello |
  6. +---------+
  7. pod "mysql-client" deleted

To demonstrate that the mysql-read Service distributes connections across servers, you can run SELECT @@server_id in a loop:

  1. kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never --\
  2. bash -ic "while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done"

You should see the reported @@server_id change randomly, because a different endpoint might be selected upon each connection attempt:

  1. +-------------+---------------------+
  2. | @@server_id | NOW() |
  3. +-------------+---------------------+
  4. | 100 | 2006-01-02 15:04:05 |
  5. +-------------+---------------------+
  6. +-------------+---------------------+
  7. | @@server_id | NOW() |
  8. +-------------+---------------------+
  9. | 102 | 2006-01-02 15:04:06 |
  10. +-------------+---------------------+
  11. +-------------+---------------------+
  12. | @@server_id | NOW() |
  13. +-------------+---------------------+
  14. | 101 | 2006-01-02 15:04:07 |
  15. +-------------+---------------------+

You can press Ctrl+C when you want to stop the loop, but it’s useful to keep it running in another window so you can see the effects of the following steps.

Simulate Pod and Node failure

To demonstrate the increased availability of reading from the pool of replicas instead of a single server, keep the SELECT @@server_id loop from above running while you force a Pod out of the Ready state.

Break the Readiness probe

The readiness probe for the mysql container runs the command mysql -h 127.0.0.1 -e 'SELECT 1' to make sure the server is up and able to execute queries.

One way to force this readiness probe to fail is to break that command:

  1. kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off

This reaches into the actual container’s filesystem for Pod mysql-2 and renames the mysql command so the readiness probe can’t find it. After a few seconds, the Pod should report one of its containers as not Ready, which you can check by running:

  1. kubectl get pod mysql-2

Look for 1/2 in the READY column:

  1. NAME READY STATUS RESTARTS AGE
  2. mysql-2 1/2 Running 0 3m

At this point, you should see your SELECT @@server_id loop continue to run, although it never reports 102 anymore. Recall that the init-mysql script defined server-id as 100 + $ordinal, so server ID 102 corresponds to Pod mysql-2.

Now repair the Pod and it should reappear in the loop output after a few seconds:

  1. kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql

Delete Pods

The StatefulSet also recreates Pods if they’re deleted, similar to what a ReplicaSet does for stateless Pods.

  1. kubectl delete pod mysql-2

The StatefulSet controller notices that no mysql-2 Pod exists anymore, and creates a new one with the same name and linked to the same PersistentVolumeClaim. You should see server ID 102 disappear from the loop output for a while and then return on its own.

Drain a Node

If your Kubernetes cluster has multiple Nodes, you can simulate Node downtime (such as when Nodes are upgraded) by issuing a drain.

First determine which Node one of the MySQL Pods is on:

  1. kubectl get pod mysql-2 -o wide

The Node name should show up in the last column:

  1. NAME READY STATUS RESTARTS AGE IP NODE
  2. mysql-2 2/2 Running 0 15m 10.244.5.27 kubernetes-node-9l2t

Then, drain the Node by running the following command, which cordons it so no new Pods may schedule there, and then evicts any existing Pods. Replace <node-name> with the name of the Node you found in the last step.

Caution: Draining a Node can impact other workloads and applications running on the same node. Only perform the following step in a test cluster.

  1. # See above advice about impact on other workloads
  2. kubectl drain <node-name> --force --delete-emptydir-data --ignore-daemonsets

Now you can watch as the Pod reschedules on a different Node:

  1. kubectl get pod mysql-2 -o wide --watch

It should look something like this:

  1. NAME READY STATUS RESTARTS AGE IP NODE
  2. mysql-2 2/2 Terminating 0 15m 10.244.1.56 kubernetes-node-9l2t
  3. [...]
  4. mysql-2 0/2 Pending 0 0s <none> kubernetes-node-fjlm
  5. mysql-2 0/2 Init:0/2 0 0s <none> kubernetes-node-fjlm
  6. mysql-2 0/2 Init:1/2 0 20s 10.244.5.32 kubernetes-node-fjlm
  7. mysql-2 0/2 PodInitializing 0 21s 10.244.5.32 kubernetes-node-fjlm
  8. mysql-2 1/2 Running 0 22s 10.244.5.32 kubernetes-node-fjlm
  9. mysql-2 2/2 Running 0 30s 10.244.5.32 kubernetes-node-fjlm

And again, you should see server ID 102 disappear from the SELECT @@server_id loop output for a while and then return.

Now uncordon the Node to return it to a normal state:

  1. kubectl uncordon <node-name>

Scaling the number of replicas

When you use MySQL replication, you can scale your read query capacity by adding replicas. For a StatefulSet, you can achieve this with a single command:

  1. kubectl scale statefulset mysql --replicas=5

Watch the new Pods come up by running:

  1. kubectl get pods -l app=mysql --watch

Once they’re up, you should see server IDs 103 and 104 start appearing in the SELECT @@server_id loop output.

You can also verify that these new servers have the data you added before they existed:

  1. kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\
  2. mysql -h mysql-3.mysql -e "SELECT * FROM test.messages"
  1. Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false
  2. +---------+
  3. | message |
  4. +---------+
  5. | hello |
  6. +---------+
  7. pod "mysql-client" deleted

Scaling back down is also seamless:

  1. kubectl scale statefulset mysql --replicas=3

Note:

Although scaling up creates new PersistentVolumeClaims automatically, scaling down does not automatically delete these PVCs.

This gives you the choice to keep those initialized PVCs around to make scaling back up quicker, or to extract data before deleting them.

You can see this by running:

  1. kubectl get pvc -l app=mysql

Which shows that all 5 PVCs still exist, despite having scaled the StatefulSet down to 3:

  1. NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
  2. data-mysql-0 Bound pvc-8acbf5dc-b103-11e6-93fa-42010a800002 10Gi RWO 20m
  3. data-mysql-1 Bound pvc-8ad39820-b103-11e6-93fa-42010a800002 10Gi RWO 20m
  4. data-mysql-2 Bound pvc-8ad69a6d-b103-11e6-93fa-42010a800002 10Gi RWO 20m
  5. data-mysql-3 Bound pvc-50043c45-b1c5-11e6-93fa-42010a800002 10Gi RWO 2m
  6. data-mysql-4 Bound pvc-500a9957-b1c5-11e6-93fa-42010a800002 10Gi RWO 2m

If you don’t intend to reuse the extra PVCs, you can delete them:

  1. kubectl delete pvc data-mysql-3
  2. kubectl delete pvc data-mysql-4

Cleaning up

  1. Cancel the SELECT @@server_id loop by pressing Ctrl+C in its terminal, or running the following from another terminal:

    1. kubectl delete pod mysql-client-loop --now
  2. Delete the StatefulSet. This also begins terminating the Pods.

    1. kubectl delete statefulset mysql
  3. Verify that the Pods disappear. They might take some time to finish terminating.

    1. kubectl get pods -l app=mysql

    You’ll know the Pods have terminated when the above returns:

    1. No resources found.
  4. Delete the ConfigMap, Services, and PersistentVolumeClaims.

    1. kubectl delete configmap,service,pvc -l app=mysql
  5. If you manually provisioned PersistentVolumes, you also need to manually delete them, as well as release the underlying resources. If you used a dynamic provisioner, it automatically deletes the PersistentVolumes when it sees that you deleted the PersistentVolumeClaims. Some dynamic provisioners (such as those for EBS and PD) also release the underlying resources upon deleting the PersistentVolumes.

What’s next