Docker Swarm

AttentionThis page documents an earlier version. Go to the latest (v2.1)version.

Docker includes swarm mode for natively managing a cluster of Docker Engines called a swarm. The Docker CLI can be used create a swarm, deploy application services to a swarm, and manage swarm behavior — without using any additional orchestration software. Details on how swarm mode works are available here.

This tutorial uses Docker Machine to create multiple nodes on your desktop. These nodes can even be on multiple machines on the cloud platform of your choice.

Prerequisites

Linux

macOS

  • Docker Engine 1.12 or later installed using Docker for Mac. Docker Machine is already included with Docker for Mac.

  • VirtualBox 5.2 or later for creating the swarm nodes.

Windows

As noted in Docker docs, the host on which Docker for Mac or Docker for Windows is installed does not itself participate in the swarm. The included version of Docker Machine is used to create the swarm nodes using VirtualBox (for macOS) and Hyper-V (for Windows).

1. Create swarm nodes

Following bash script is a simpler form of Docker’s own swarm beginner tutorial bash script. You can use this for Linux and macOS. If you are using Windows, then download and change the powershell Hyper-V version of the same script.

  • The script first instantiates 3 nodes using Docker Machine and VirtualBox. Thereafter, it initializes the swarm cluster by creating a swarm manager on the first node. Finally, it adds the remaining nodes as workers to the cluster. It also pulls the yugabytedb/yugabyte container image into each of the nodes to expedite the next steps.

NoteIn more fault-tolerant setups, there will be multiple manager nodes and they will be independent of the worker nodes. A 3-node master and 3-node worker setup is used in the Docker tutorial script referenced above.

  1. # !/bin/bash
  3. # Swarm mode using Docker Machine
  5. workers=3
  7. # create worker machines
  8. echo "======> Creating $workers worker machines ...";
  9. for node in $(seq 1 $workers);
  10. do
  11.     echo "======> Creating worker$node machine ...";
  12.     docker-machine create -d virtualbox worker$node;
  13. done
  15. # list all machines
  16. docker-machine ls
  18. # initialize swarm mode and create a manager on worker1
  19. echo "======> Initializing the swarm manager on worker1 ..."
  20. docker-machine ssh worker1 "docker swarm init --listen-addr $(docker-machine ip worker1) --advertise-addr $(docker-machine ip worker1)"
  22. # get worker tokens
  23. export worker_token=`docker-machine ssh worker1 "docker swarm join-token worker -q"`
  24. echo "worker_token: $worker_token"
  26. # show members of swarm
  27. docker-machine ssh worker1 "docker node ls"
  29. # other workers join swarm, worker1 is already a member
  30. for node in $(seq 2 $workers);
  31. do
  32.     echo "======> worker$node joining swarm as worker ..."
  33.     docker-machine ssh worker$node \
  34.     "docker swarm join \
  35.     --token $worker_token \
  36.     --listen-addr $(docker-machine ip worker$node) \
  37.     --advertise-addr $(docker-machine ip worker$node) \
  38.     $(docker-machine ip worker1)"
  39. done
  41. # pull the yugabytedb container
  42. for node in $(seq 1 $workers);
  43. do
  44.     echo "======> pulling yugabytedb/yugabyte container on worker$node ..."
  45.     docker-machine ssh worker$node \
  46.     "docker pull yugabytedb/yugabyte"
  47. done
  49. # show members of swarm
  50. docker-machine ssh worker1 "docker node ls"
  • Review all the nodes created.
  1. $ docker-machine ls 
  1. NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
  2. worker1   -        virtualbox   Running   tcp://192.168.99.v1.0:2376           v18.05.0-ce   
  3. worker2   -        virtualbox   Running   tcp://192.168.99.101:2376           v18.05.0-ce   
  4. worker3   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.05.0-ce

2. Create overlay network

  • SSH into the worker1 node where the swarm manager is running.
  1. $ docker-machine ssh worker1
  • Create an overlay network that the swarm services can use to communicate with each other. The attachable option allows standalone containers to connect to swarm services on the network.

You can do this as shown below.

  1. $ docker network create --driver overlay --attachable yugabytedb

3. Create yb-master services

  • Create 3 yb-master replicated services each with replicas set to 1. This is the only way in Docker Swarm today to get stable network identies for each of yb-master containers that we will need to provide as input for creating the yb-tserver service in the next step.

Note for Kubernetes UsersDocker Swarm lacks an equivalent of Kubernetes StatefulSets. The concept of replicated services is similar to Kubernetes Deployments.

  1. $ docker service create \
  2. --replicas 1 \
  3. --name yb-master1 \
  4. --network yugabytedb \
  5. --mount type=volume,source=yb-master1,target=/mnt/data0 \
  6. --publish 7000:7000 \
  7. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
  8. --fs_data_dirs=/mnt/data0 \
  9. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
  10. --replication_factor=3
  1. $ docker service create \
  2. --replicas 1 \
  3. --name yb-master2 \
  4. --network yugabytedb \
  5. --mount type=volume,source=yb-master2,target=/mnt/data0 \
  6. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
  7. --fs_data_dirs=/mnt/data0 \
  8. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
  9. --replication_factor=3
  1. $ docker service create \
  2. --replicas 1 \
  3. --name yb-master3 \
  4. --network yugabytedb \
  5. --mount type=volume,source=yb-master3,target=/mnt/data0 \
  6. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-master \
  7. --fs_data_dirs=/mnt/data0 \
  8. --master_addresses=yb-master1:7100,yb-master2:7100,yb-master3:7100 \
  9. --replication_factor=3
  • Run the command below to see the services that are now live.
  1. $ docker service ls
  1. ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
  2. jfnrqfvnrc5b        yb-master1          replicated          1/1                 yugabytedb/yugabyte:latest   *:7000->7000/tcp
  3. kqp6eju3kq88        yb-master2          replicated          1/1                 yugabytedb/yugabyte:latest   
  4. ah6wfodd4noh        yb-master3          replicated          1/1                 yugabytedb/yugabyte:latest
  • View the yb-master admin UI by going to the port 7000 of any node, courtesy of the publish option used when yb-master1 was created. For e.g., we can see from Step 1 that worker2’s IP address is 192.168.99.101. So, http://192.168.99.101:7000 takes us to the yb-master admin UI.

4. Create yb-tserver service

  • Create a single yb-tserver global service so that swarm can then automatically spawn 1 container/task on each worker node. Each time we add a node to the swarm, the swarm orchestrator creates a task and the scheduler assigns the task to the new node.

Note for Kubernetes UsersThe global services concept in Docker Swarm is similar to Kubernetes DaemonSets.

  1. $ docker service create \
  2. --mode global \
  3. --name yb-tserver \
  4. --network yugabytedb \
  5. --mount type=volume,source=yb-tserver,target=/mnt/data0 \
  6. --publish 9000:9000 \
  7. yugabytedb/yugabyte:latest /home/yugabyte/bin/yb-tserver \
  8. --fs_data_dirs=/mnt/data0 \
  9. --tserver_master_addrs=yb-master1:7100,yb-master2:7100,yb-master3:7100

TipUse remote volumes instead of local volumes (used above) when you want to scale-out or scale-in your swarm cluster.

  • Run the command below to see the services that are now live.
  1. $ docker service ls
  1. ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
  2. jfnrqfvnrc5b        yb-master1          replicated          1/1                 yugabytedb/yugabyte:latest   *:7000->7000/tcp
  3. kqp6eju3kq88        yb-master2          replicated          1/1                 yugabytedb/yugabyte:latest   
  4. ah6wfodd4noh        yb-master3          replicated          1/1                 yugabytedb/yugabyte:latest   
  5. n6padh2oqjk7        yb-tserver          global              3/3                 yugabytedb/yugabyte:latest   *:9000->9000/tcp

5. Test the client APIs

YCQL API

  • Find the container ID of the yb-tserver running on worker1. Use the first param of docker ps output.

  • Connect to that container using that container ID.

  1. $ docker exec -it <ybtserver_container_id> /home/yugabyte/bin/cqlsh
  1. Connected to local cluster at 127.0.0.1:9042.
  2. [cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
  3. Use HELP for help.
  4. cqlsh>

YEDIS API

  • Find the container ID of the yb-master running on worker1. Use the first param of docker ps output.

  • Initialize the YEDIS API.

You can do this as shown below.

  1. $ docker exec -it <ybmaster_container_id> /home/yugabyte/bin/yb-admin -- --master_addresses yb-master1:7100,yb-master2:7100,yb-master3:7100 setup_redis_table
  1. ...
  2. I0515 19:54:48.952378    39 client.cc:1208] Created table system_redis.redis of type REDIS_TABLE_TYPE
  3. I0515 19:54:48.953572    39 yb-admin_client.cc:440] Table 'system_redis.redis' created.

PostgreSQL API

  • Install the postgresql client in the yb-tserver container.
  1. $ docker exec -it <ybtserver_container_id> yum install postgresql
  • Connect to the psql client in yb-tserver.
  1. $ docker exec -it <ybtserver_container_id> psql -h localhost --port 5433
  1. ...
  2. psql (9.2.23, server 0.0.0)
  3. WARNING: psql version 9.2, server version 0.0.
  4.      Some psql features might not work.
  5. Type "help" for help.
  7. root=>

6. Test fault-tolerance with node failure

Docker Swarm ensures that the yb-tserver global service will always have 1 yb-tserver container running on every node. If the yb-tserver container on any node dies, then Docker Swarm will bring it back on.

  1. $ docker kill <ybtserver_container_id>

Observe the output of docker ps every few seconds till you see that the yb-tserver container is re-spawned by Docker Swarm.

7. Test auto-scaling with node addition

  • On the host machine, get worker token for new worker nodes to use to join the existing swarm.
  1. $ docker-machine ssh worker1 "docker swarm join-token worker -q"
  1. SWMTKN-1-aadasdsadas-2ja2q2esqsivlfx2ygi8u62yq
  • Create a new node worker4.
  1. $ docker-machine create -d virtualbox worker4
  • Pull the YugabyteDB container.
  1. $ docker-machine ssh worker4 "docker pull yugabytedb/yugabyte"
  • Join worker4 with existing swarm.
  1. $ docker-machine ssh worker4 \
  2.     "docker swarm join \
  3.     --token SWMTKN-1-aadasdsadas-2ja2q2esqsivlfx2ygi8u62yq \
  4.     --listen-addr $(docker-machine ip worker4) \
  5.     --advertise-addr $(docker-machine ip worker4) \
  6.     $(docker-machine ip worker1)"
  • Observe that Docker Swarm adds a new yb-tserver instance to the newly added worker4 node and changes its replica status from 3 / 3 to 4 / 4.

You can do this as shown below.

  1. $ docker service ls
  1. ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
  2. jfnrqfvnrc5b        yb-master1          replicated          1/1                 yugabytedb/yugabyte:latest   *:7000->7000/tcp
  3. kqp6eju3kq88        yb-master2          replicated          1/1                 yugabytedb/yugabyte:latest   
  4. ah6wfodd4noh        yb-master3          replicated          1/1                 yugabytedb/yugabyte:latest   
  5. n6padh2oqjk7        yb-tserver          global              4/4                 yugabytedb/yugabyte:latest   *:9000->9000/tcp

8. Remove services and destroy nodes

  • Stop the machines.
  1. $ docker-machine stop $(docker-machine ls -q)
  • Remove the machines.
  1. $ docker-machine rm $(docker-machine ls -q)