Demo

Procedures for working with an etcd cluster

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

01_etcd_clustering_2016050601

On each etcd node, specify the cluster members:

  1. TOKEN=token-01
  2. CLUSTER_STATE=new
  3. NAME_1=machine-1
  4. NAME_2=machine-2
  5. NAME_3=machine-3
  6. HOST_1=10.240.0.17
  7. HOST_2=10.240.0.18
  8. HOST_3=10.240.0.19
  9. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

  1. # For machine 1
  2. THIS_NAME=${NAME_1}
  3. THIS_IP=${HOST_1}
  4. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  5. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  6. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  7. --initial-cluster ${CLUSTER} \
  8. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  9. # For machine 2
  10. THIS_NAME=${NAME_2}
  11. THIS_IP=${HOST_2}
  12. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  13. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  14. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  15. --initial-cluster ${CLUSTER} \
  16. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  17. # For machine 3
  18. THIS_NAME=${NAME_3}
  19. THIS_IP=${HOST_3}
  20. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  21. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  22. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  23. --initial-cluster ${CLUSTER} \
  24. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

  1. curl https://discovery.etcd.io/new?size=3
  2. https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92
  3. # grab this token
  4. TOKEN=token-01
  5. CLUSTER_STATE=new
  6. NAME_1=machine-1
  7. NAME_2=machine-2
  8. NAME_3=machine-3
  9. HOST_1=10.240.0.17
  10. HOST_2=10.240.0.18
  11. HOST_3=10.240.0.19
  12. DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92
  13. THIS_NAME=${NAME_1}
  14. THIS_IP=${HOST_1}
  15. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  16. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  17. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  18. --discovery ${DISCOVERY} \
  19. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  20. THIS_NAME=${NAME_2}
  21. THIS_IP=${HOST_2}
  22. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  23. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  24. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  25. --discovery ${DISCOVERY} \
  26. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
  27. THIS_NAME=${NAME_3}
  28. THIS_IP=${HOST_3}
  29. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  30. --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  31. --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  32. --discovery ${DISCOVERY} \
  33. --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Now etcd is ready! To connect to etcd with etcdctl:

  1. export ETCDCTL_API=3
  2. HOST_1=10.240.0.17
  3. HOST_2=10.240.0.18
  4. HOST_3=10.240.0.19
  5. ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
  6. etcdctl --endpoints=$ENDPOINTS member list

Access etcd

02_etcdctl_access_etcd_2016051001

put command to write:

  1. etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

get to read from etcd:

  1. etcdctl --endpoints=$ENDPOINTS get foo
  2. etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo

Get by prefix

03_etcdctl_get_by_prefix_2016050501

  1. etcdctl --endpoints=$ENDPOINTS put web1 value1
  2. etcdctl --endpoints=$ENDPOINTS put web2 value2
  3. etcdctl --endpoints=$ENDPOINTS put web3 value3
  4. etcdctl --endpoints=$ENDPOINTS get web --prefix

Delete

04_etcdctl_delete_2016050601

  1. etcdctl --endpoints=$ENDPOINTS put key myvalue
  2. etcdctl --endpoints=$ENDPOINTS del key
  3. etcdctl --endpoints=$ENDPOINTS put k1 value1
  4. etcdctl --endpoints=$ENDPOINTS put k2 value2
  5. etcdctl --endpoints=$ENDPOINTS del k --prefix

Transactional write

txn to wrap multiple requests into one transaction:

05_etcdctl_transaction_2016050501

  1. etcdctl --endpoints=$ENDPOINTS put user1 bad
  2. etcdctl --endpoints=$ENDPOINTS txn --interactive
  3. compares:
  4. value("user1") = "bad"
  5. success requests (get, put, delete):
  6. del user1
  7. failure requests (get, put, delete):
  8. put user1 good

Watch

watch to get notified of future changes:

06_etcdctl_watch_2016050501

  1. etcdctl --endpoints=$ENDPOINTS watch stock1
  2. etcdctl --endpoints=$ENDPOINTS put stock1 1000
  3. etcdctl --endpoints=$ENDPOINTS watch stock --prefix
  4. etcdctl --endpoints=$ENDPOINTS put stock1 10
  5. etcdctl --endpoints=$ENDPOINTS put stock2 20

Lease

lease to write with TTL:

07_etcdctl_lease_2016050501

  1. etcdctl --endpoints=$ENDPOINTS lease grant 300
  2. # lease 2be7547fbc6a5afa granted with TTL(300s)
  3. etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
  4. etcdctl --endpoints=$ENDPOINTS get sample
  5. etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
  6. etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
  7. # or after 300 seconds
  8. etcdctl --endpoints=$ENDPOINTS get sample

Distributed locks

lock for distributed lock:

08_etcdctl_lock_2016050501

  1. etcdctl --endpoints=$ENDPOINTS lock mutex1
  2. # another client with the same name blocks
  3. etcdctl --endpoints=$ENDPOINTS lock mutex1

Elections

elect for leader election:

09_etcdctl_elect_2016050501

  1. etcdctl --endpoints=$ENDPOINTS elect one p1
  2. # another client with the same name blocks
  3. etcdctl --endpoints=$ENDPOINTS elect one p2

Cluster status

Specify the initial cluster configuration for each machine:

10_etcdctl_endpoint_2016050501

  1. etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status
  2. +------------------+------------------+---------+---------+-----------+-----------+------------+
  3. | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
  4. +------------------+------------------+---------+---------+-----------+-----------+------------+
  5. | 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.0.0 | 45 kB | true | 4 | 16726 |
  6. | 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.0.0 | 45 kB | false | 4 | 16726 |
  7. | 10.240.0.19:2379 | 94df724b66343e6c | 3.0.0 | 45 kB | false | 4 | 16726 |
  8. +------------------+------------------+---------+---------+-----------+-----------+------------+
  1. etcdctl --endpoints=$ENDPOINTS endpoint health
  2. 10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
  3. 10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
  4. 10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

Snapshot

snapshot to save point-in-time snapshot of etcd database:

11_etcdctl_snapshot_2016051001

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

  1. ENDPOINTS=$HOST_1:2379
  2. etcdctl --endpoints=$ENDPOINTS snapshot save my.db
  3. Snapshot saved at my.db
  1. etcdctl --write-out=table --endpoints=$ENDPOINTS snapshot status my.db
  2. +---------+----------+------------+------------+
  3. | HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
  4. +---------+----------+------------+------------+
  5. | c55e8b8 | 9 | 13 | 25 kB |
  6. +---------+----------+------------+------------+

Migrate

migrate to transform etcd v2 to v3 data:

12_etcdctl_migrate_2016061602

  1. # write key in etcd version 2 store
  2. export ETCDCTL_API=2
  3. etcdctl --endpoints=http://$ENDPOINT set foo bar
  4. # read key in etcd v2
  5. etcdctl --endpoints=$ENDPOINTS --output="json" get foo
  6. # stop etcd node to migrate, one by one
  7. # migrate v2 data
  8. export ETCDCTL_API=3
  9. etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal"
  10. # restart etcd node after migrate, one by one
  11. # confirm that the key got migrated
  12. etcdctl --endpoints=$ENDPOINTS get /foo

Member

member to add,remove,update membership:

13_etcdctl_member_2016062301

  1. # For each machine
  2. TOKEN=my-etcd-token-1
  3. CLUSTER_STATE=new
  4. NAME_1=etcd-node-1
  5. NAME_2=etcd-node-2
  6. NAME_3=etcd-node-3
  7. HOST_1=10.240.0.13
  8. HOST_2=10.240.0.14
  9. HOST_3=10.240.0.15
  10. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
  11. # For node 1
  12. THIS_NAME=${NAME_1}
  13. THIS_IP=${HOST_1}
  14. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  15. --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  16. --listen-peer-urls http://${THIS_IP}:2380 \
  17. --advertise-client-urls http://${THIS_IP}:2379 \
  18. --listen-client-urls http://${THIS_IP}:2379 \
  19. --initial-cluster ${CLUSTER} \
  20. --initial-cluster-state ${CLUSTER_STATE} \
  21. --initial-cluster-token ${TOKEN}
  22. # For node 2
  23. THIS_NAME=${NAME_2}
  24. THIS_IP=${HOST_2}
  25. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  26. --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  27. --listen-peer-urls http://${THIS_IP}:2380 \
  28. --advertise-client-urls http://${THIS_IP}:2379 \
  29. --listen-client-urls http://${THIS_IP}:2379 \
  30. --initial-cluster ${CLUSTER} \
  31. --initial-cluster-state ${CLUSTER_STATE} \
  32. --initial-cluster-token ${TOKEN}
  33. # For node 3
  34. THIS_NAME=${NAME_3}
  35. THIS_IP=${HOST_3}
  36. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  37. --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  38. --listen-peer-urls http://${THIS_IP}:2380 \
  39. --advertise-client-urls http://${THIS_IP}:2379 \
  40. --listen-client-urls http://${THIS_IP}:2379 \
  41. --initial-cluster ${CLUSTER} \
  42. --initial-cluster-state ${CLUSTER_STATE} \
  43. --initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

  1. # get member ID
  2. export ETCDCTL_API=3
  3. HOST_1=10.240.0.13
  4. HOST_2=10.240.0.14
  5. HOST_3=10.240.0.15
  6. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list
  7. # remove the member
  8. MEMBER_ID=278c654c9a6dfd3b
  9. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
  10. member remove ${MEMBER_ID}
  11. # add a new member (node 4)
  12. export ETCDCTL_API=3
  13. NAME_1=etcd-node-1
  14. NAME_2=etcd-node-2
  15. NAME_4=etcd-node-4
  16. HOST_1=10.240.0.13
  17. HOST_2=10.240.0.14
  18. HOST_4=10.240.0.16 # new member
  19. etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
  20. member add ${NAME_4} \
  21. --peer-urls=http://${HOST_4}:2380

Next, start the new member with --initial-cluster-state existing flag:

  1. # [WARNING] If the new member starts from the same disk space,
  2. # make sure to remove the data directory of the old member
  3. #
  4. # restart with 'existing' flag
  5. TOKEN=my-etcd-token-1
  6. CLUSTER_STATE=existing
  7. NAME_1=etcd-node-1
  8. NAME_2=etcd-node-2
  9. NAME_4=etcd-node-4
  10. HOST_1=10.240.0.13
  11. HOST_2=10.240.0.14
  12. HOST_4=10.240.0.16 # new member
  13. CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380
  14. THIS_NAME=${NAME_4}
  15. THIS_IP=${HOST_4}
  16. etcd --data-dir=data.etcd --name ${THIS_NAME} \
  17. --initial-advertise-peer-urls http://${THIS_IP}:2380 \
  18. --listen-peer-urls http://${THIS_IP}:2380 \
  19. --advertise-client-urls http://${THIS_IP}:2379 \
  20. --listen-client-urls http://${THIS_IP}:2379 \
  21. --initial-cluster ${CLUSTER} \
  22. --initial-cluster-state ${CLUSTER_STATE} \
  23. --initial-cluster-token ${TOKEN}

Auth

auth,user,role for authentication:

14_etcdctl_auth_2016062301

  1. export ETCDCTL_API=3
  2. ENDPOINTS=localhost:2379
  3. etcdctl --endpoints=${ENDPOINTS} role add root
  4. etcdctl --endpoints=${ENDPOINTS} role grant-permission root readwrite foo
  5. etcdctl --endpoints=${ENDPOINTS} role get root
  6. etcdctl --endpoints=${ENDPOINTS} user add root
  7. etcdctl --endpoints=${ENDPOINTS} user grant-role root root
  8. etcdctl --endpoints=${ENDPOINTS} user get root
  9. etcdctl --endpoints=${ENDPOINTS} auth enable
  10. # now all client requests go through auth
  11. etcdctl --endpoints=${ENDPOINTS} --user=root:123 put foo bar
  12. etcdctl --endpoints=${ENDPOINTS} get foo
  13. etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo
  14. etcdctl --endpoints=${ENDPOINTS} --user=root:123 get foo1

Last modified June 14, 2021: Renaming content/en/next folder to content/en/v3.5. Updating redirects, links, and config as needed. (#363) (138926b)