Recommended etcd practices

This topic provides recommended performance and scalability practices for etcd in OKD.

Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance. Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because etcd’s consensus protocol depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.

Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure.

In terms of latency, run etcd on top of a block device that can write at least 50 IOPS of 8000 bytes long sequentially. That is, with a latency of 10ms, keep in mind that uses fdatasync to synchronize each write in the WAL. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as fio.

To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.

The load on etcd arises from static factors, such as the number of nodes and pods, and dynamic factors, including changes in endpoints due to pod autoscaling, pod restarts, job executions, and other workload-related events. To accurately size your etcd setup, you must analyze the specific requirements of your workload. Consider the number of nodes, pods, and other relevant factors that impact the load on etcd.

The following hard disk features provide optimal etcd performance:

  • Low latency to support fast read operation.

  • High-bandwidth writes for faster compactions and defragmentation.

  • High-bandwidth reads for faster recovery from failures.

  • Solid state drives as a minimum selection, however NVMe drives are preferred.

  • Server-grade hardware from various manufacturers for increased reliability.

  • RAID 0 technology for increased performance.

  • Dedicated etcd drives. Do not place log files or other heavy workloads on etcd drives.

Avoid NAS or SAN setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use PCI passthrough to pass NVM devices directly to the nodes.

Always benchmark by using utilities such as fio. You can use such utilities to continuously monitor the cluster performance as it increases.

Avoid using the Network File System (NFS) protocol or other network based file systems.

Some key metrics to monitor on a deployed OKD cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics.

The etcd member database sizes can vary in a cluster during normal operations. This difference does not affect cluster upgrades, even if the leader size is different from the other members.

To validate the hardware for etcd before or after you create the OKD cluster, you can use fio.

Prerequisites

  • Container runtimes such as Podman or Docker are installed on the machine that you’re testing.

  • Data is written to the /var/lib/etcd path.

Procedure

  • Run fio and analyze the results:

    • If you use Podman, run this command:

      1. $ sudo podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf

    • If you use Docker, run this command:

      1. $ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf

The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms. A few of the most important etcd metrics that might affected by I/O performance are as follow:

  • etcd_disk_wal_fsync_duration_seconds_bucket metric reports the etcd’s WAL fsync duration

  • etcd_disk_backend_commit_duration_seconds_bucket metric reports the etcd backend commit latency duration

  • etcd_server_leader_changes_seen_total metric reports the leader changes

Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed OKD cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.

The histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m])) metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.

Additional resources

Moving etcd to a different disk

You can move etcd from a shared disk to a separate disk to prevent or resolve performance issues.

Prerequisites

  • The MachineConfigPool must match metadata.labels[machineconfiguration.openshift.io/role]. This applies to a controller, worker, or a custom pool.

  • The node’s auxiliary storage device, such as /dev/sdb, must match the sdb. Change this reference in all places in the file.

This procedure does not move parts of the root file system, such as /var/, to another disk or partition on an installed node.

The Machine Config Operator (MCO) is responsible for mounting a secondary disk for an OKD 4.14 container storage.

Use the following steps to move etcd to a different device:

Procedure

  1. Create a machineconfig YAML file named etcd-mc.yml and add the following information:

    1. apiVersion: machineconfiguration.openshift.io/v1
    2. kind: MachineConfig
    3. metadata:
    4.   labels:
    5.     machineconfiguration.openshift.io/role: master
    6.   name: 98-var-lib-etcd
    7. spec:
    8.   config:
    9.     ignition:
    10.       version: 3.2.0
    11.     systemd:
    12.       units:
    13.       - contents: |
    14.           [Unit]
    15.           Description=Make File System on /dev/sdb
    16.           DefaultDependencies=no
    17.           BindsTo=dev-sdb.device
    18.           After=dev-sdb.device var.mount
    19.           Before=systemd-fsck@dev-sdb.service
    21.           [Service]
    22.           Type=oneshot
    23.           RemainAfterExit=yes
    24.           ExecStart=/usr/lib/systemd/systemd-makefs xfs /dev/sdb
    25.           TimeoutSec=0
    27.           [Install]
    28.           WantedBy=var-lib-containers.mount
    29.         enabled: true
    30.         name: systemd-mkfs@dev-sdb.service
    31.       - contents: |
    32.           [Unit]
    33.           Description=Mount /dev/sdb to /var/lib/etcd
    34.           Before=local-fs.target
    35.           Requires=systemd-mkfs@dev-sdb.service
    36.           After=systemd-mkfs@dev-sdb.service var.mount
    38.           [Mount]
    39.           What=/dev/sdb
    40.           Where=/var/lib/etcd
    41.           Type=xfs
    42.           Options=defaults,prjquota
    44.           [Install]
    45.           WantedBy=local-fs.target
    46.         enabled: true
    47.         name: var-lib-etcd.mount
    48.       - contents: |
    49.           [Unit]
    50.           Description=Sync etcd data if new mount is empty
    51.           DefaultDependencies=no
    52.           After=var-lib-etcd.mount var.mount
    53.           Before=crio.service
    55.           [Service]
    56.           Type=oneshot
    57.           RemainAfterExit=yes
    58.           ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
    59.           ExecStart=/usr/sbin/setenforce 0
    60.           ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
    61.           ExecStart=/usr/sbin/setenforce 1
    62.           TimeoutSec=0
    64.           [Install]
    65.           WantedBy=multi-user.target graphical.target
    66.         enabled: true
    67.         name: sync-var-lib-etcd-to-etcd.service
    68.       - contents: |
    69.           [Unit]
    70.           Description=Restore recursive SELinux security contexts
    71.           DefaultDependencies=no
    72.           After=var-lib-etcd.mount
    73.           Before=crio.service
    75.           [Service]
    76.           Type=oneshot
    77.           RemainAfterExit=yes
    78.           ExecStart=/sbin/restorecon -R /var/lib/etcd/
    79.           TimeoutSec=0
    81.           [Install]
    82.           WantedBy=multi-user.target graphical.target
    83.         enabled: true
    84.         name: restorecon-var-lib-etcd.service

  2. Create the machine configuration by entering the following commands:

    1. $ oc login -u ${ADMIN} -p ${ADMINPASSWORD} ${API}
    2. ... output omitted ...
    1. $ oc create -f etcd-mc.yml
    2. machineconfig.machineconfiguration.openshift.io/98-var-lib-etcd created
    1. $ oc login -u ${ADMIN} -p ${ADMINPASSWORD} ${API}
    2.  [... output omitted ...]
    1. $ oc create -f etcd-mc.yml machineconfig.machineconfiguration.openshift.io/98-var-lib-etcd created

    The nodes are updated and rebooted. After the reboot completes, the following events occur:

    • An XFS file system is created on the specified disk.

    • The disk mounts to /var/lib/etc.

    • The content from /sysroot/ostree/deploy/rhcos/var/lib/etcd syncs to /var/lib/etcd.

    • A restore of SELinux labels is forced for /var/lib/etcd.

    • The old content is not removed.

  3. After the nodes are on a separate disk, update the machine configuration file, etcd-mc.yml with the following information:

    1. apiVersion: machineconfiguration.openshift.io/v1
    2. kind: MachineConfig
    3. metadata:
    4.   labels:
    5.     machineconfiguration.openshift.io/role: master
    6.   name: 98-var-lib-etcd
    7. spec:
    8.   config:
    9.     ignition:
    10.       version: 3.2.0
    11.     systemd:
    12.       units:
    13.       - contents: |
    14.           [Unit]
    15.           Description=Mount /dev/sdb to /var/lib/etcd
    16.           Before=local-fs.target
    17.           Requires=systemd-mkfs@dev-sdb.service
    18.           After=systemd-mkfs@dev-sdb.service var.mount
    20.           [Mount]
    21.           What=/dev/sdb
    22.           Where=/var/lib/etcd
    23.           Type=xfs
    24.           Options=defaults,prjquota
    26.           [Install]
    27.           WantedBy=local-fs.target
    28.         enabled: true
    29.         name: var-lib-etcd.mount

  4. Apply the modified version that removes the logic for creating and syncing the device by entering the following command:

    1. $ oc replace -f etcd-mc.yml

    The previous step prevents the nodes from rebooting.

Additional resources

Defragmenting etcd data

For large and dense clusters, etcd can suffer from poor performance if the keyspace grows too large and exceeds the space quota. Periodically maintain and defragment etcd to free up space in the data store. Monitor Prometheus for etcd metrics and defragment it when required; otherwise, etcd can raise a cluster-wide alarm that puts the cluster into a maintenance mode that accepts only key reads and deletes.

Monitor these key metrics:

  • etcd_server_quota_backend_bytes, which is the current quota limit

  • etcd_mvcc_db_total_size_in_use_in_bytes, which indicates the actual database usage after a history compaction

  • etcd_mvcc_db_total_size_in_bytes, which shows the database size, including free space waiting for defragmentation

Defragment etcd data to reclaim disk space after events that cause disk fragmentation, such as etcd history compaction.

History compaction is performed automatically every five minutes and leaves gaps in the back-end database. This fragmented space is available for use by etcd, but is not available to the host file system. You must defragment etcd to make this space available to the host file system.

Defragmentation occurs automatically, but you can also trigger it manually.

Automatic defragmentation is good for most cases, because the etcd operator uses cluster information to determine the most efficient operation for the user.

Automatic defragmentation

The etcd Operator automatically defragments disks. No manual intervention is needed.

Verify that the defragmentation process is successful by viewing one of these logs:

  • etcd logs

  • cluster-etcd-operator pod

  • operator status error log

Automatic defragmentation can cause leader election failure in various OpenShift core components, such as the Kubernetes controller manager, which triggers a restart of the failing component. The restart is harmless and either triggers failover to the next running instance or the component resumes work again after the restart.

Example log output for successful defragmentation

  1. etcd member has been defragmented: <member_name>, memberID: <member_id>

Example log output for unsuccessful defragmentation

  1. failed defrag on member: <member_name>, memberID: <member_id>: <error_message>

Manual defragmentation

A Prometheus alert indicates when you need to use manual defragmentation. The alert is displayed in two cases:

  • When etcd uses more than 50% of its available space for more than 10 minutes

  • When etcd is actively using less than 50% of its total database size for more than 10 minutes

You can also determine whether defragmentation is needed by checking the etcd database size in MB that will be freed by defragmentation with the PromQL expression: (etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes)/1024/1024

Defragmenting etcd is a blocking action. The etcd member will not respond until defragmentation is complete. For this reason, wait at least one minute between defragmentation actions on each of the pods to allow the cluster to recover.

Follow this procedure to defragment etcd data on each etcd member.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Determine which etcd member is the leader, because the leader should be defragmented last.

    1. Get the list of etcd pods:

      1. $ oc -n openshift-etcd get pods -l k8s-app=etcd -o wide

      Example output

      1. etcd-ip-10-0-159-225.example.redhat.com                3/3     Running     0          175m   10.0.159.225   ip-10-0-159-225.example.redhat.com   <none>           <none>
      2. etcd-ip-10-0-191-37.example.redhat.com                 3/3     Running     0          173m   10.0.191.37    ip-10-0-191-37.example.redhat.com    <none>           <none>
      3. etcd-ip-10-0-199-170.example.redhat.com                3/3     Running     0          176m   10.0.199.170   ip-10-0-199-170.example.redhat.com   <none>           <none>

    2. Choose a pod and run the following command to determine which etcd member is the leader:

      1. $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.example.redhat.com etcdctl endpoint status --cluster -w table

      Example output

      1. Defaulting container name to etcdctl.
      2. Use 'oc describe pod/etcd-ip-10-0-159-225.example.redhat.com -n openshift-etcd' to see all of the containers in this pod.
      3. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      4. |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      5. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      6. |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.5.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      7. | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.5.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      8. | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.5.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      9. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      Based on the IS LEADER column of this output, the https://10.0.199.170:2379 endpoint is the leader. Matching this endpoint with the output of the previous step, the pod name of the leader is etcd-ip-10-0-199-170.example.redhat.com.

  2. Defragment an etcd member.

    1. Connect to the running etcd container, passing in the name of a pod that is not the leader:

      1. $ oc rsh -n openshift-etcd etcd-ip-10-0-159-225.example.redhat.com

    2. Unset the ETCDCTL_ENDPOINTS environment variable:

      1. sh-4.4# unset ETCDCTL_ENDPOINTS

    3. Defragment the etcd member:

      1. sh-4.4# etcdctl --command-timeout=30s --endpoints=https://localhost:2379 defrag

      Example output

      1. Finished defragmenting etcd member[https://localhost:2379]

      If a timeout error occurs, increase the value for --command-timeout until the command succeeds.

    4. Verify that the database size was reduced:

      1. sh-4.4# etcdctl endpoint status -w table --cluster

      Example output

      1. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      2. |         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
      3. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
      4. |  https://10.0.191.37:2379 | 251cd44483d811c3 |   3.5.9 |  104 MB |     false |      false |         7 |      91624 |              91624 |        |
      5. | https://10.0.159.225:2379 | 264c7c58ecbdabee |   3.5.9 |   41 MB |     false |      false |         7 |      91624 |              91624 |        | (1)
      6. | https://10.0.199.170:2379 | 9ac311f93915cc79 |   3.5.9 |  104 MB |      true |      false |         7 |      91624 |              91624 |        |
      7. +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

      This example shows that the database size for this etcd member is now 41 MB as opposed to the starting size of 104 MB.

    5. Repeat these steps to connect to each of the other etcd members and defragment them. Always defragment the leader last.

      Wait at least one minute between defragmentation actions to allow the etcd pod to recover. Until the etcd pod recovers, the etcd member will not respond.

  3. If any NOSPACE alarms were triggered due to the space quota being exceeded, clear them.

    1. Check if there are any NOSPACE alarms:

      1. sh-4.4# etcdctl alarm list

      Example output

      1. memberID:12345678912345678912 alarm:NOSPACE

    2. Clear the alarms:

      1. sh-4.4# etcdctl alarm disarm

Setting tuning parameters for etcd

You can set the control plane hardware speed to "Standard", "Slower", or the default, which is "".

The default setting allows the system to decide which speed to use. This value enables upgrades from versions where this feature does not exist, as the system can select values from previous versions.

By selecting one of the other values, you are overriding the default. If you see many leader elections due to timeouts or missed heartbeats and your system is set to "" or "Standard", set the hardware speed to "Slower" to make the system more tolerant to the increased latency.

Tuning etcd latency tolerances is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Changing hardware speed tolerance

To change the hardware speed tolerance for etcd, complete the following steps.

Procedure

  1. Check to see what the current value is by entering the following command:

    1. $ oc describe etcd/cluster | grep "Control Plane Hardware Speed"

    Example output

    1. Control Plane Hardware Speed:  <VALUE>

    If the output is empty, the field has not been set and should be considered as the default (“”).

  2. Change the value by entering the following command. Replace <value> with one of the valid values: "", "Standard", or "Slower":

    1. oc patch etcd/cluster --type=merge -p '{"spec": {"controlPlaneHardwareSpeed": "<value>"}}'

    The following table indicates the heartbeat interval and leader election timeout for each profile. These values are subject to change.

    Profile

    ETCD_HEARTBEAT_INTERVAL

    ETCD_LEADER_ELECTION_TIMEOUT

    “”

    Varies depending on platform

    Varies depending on platform

    Standard

    100

    1000

    Slower

    500

    2500

  3. Review the output:

    Example output

    1. etcd.operator.openshift.io/cluster patched

    If you enter any value besides the valid values, error output is displayed. For example, if you entered "Faster" as the value, the output is as follows:

    Example output

    1. The Etcd "cluster" is invalid: spec.controlPlaneHardwareSpeed: Unsupported value: "Faster": supported values: "", "Standard", "Slower"

  4. Verify that the value was changed by entering the following command:

    1. $ oc describe etcd/cluster | grep "Control Plane Hardware Speed"

    Example output

    1. Control Plane Hardware Speed:  ""

  5. Wait for etcd pods to roll out:

    1. oc get pods -n openshift-etcd -w

    The following output shows the expected entries for master-0. Before you continue, wait until all masters show a status of 4/4 Running.

    Example output

    1. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     Pending             0          0s
    2. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     Pending             0          0s
    3. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     ContainerCreating   0          0s
    4. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     ContainerCreating   0          1s
    5. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           1/1     Running             0          2s
    6. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     Completed           0          34s
    7. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     Completed           0          36s
    8. installer-9-ci-ln-qkgs94t-72292-9clnd-master-0           0/1     Completed           0          36s
    9. etcd-guard-ci-ln-qkgs94t-72292-9clnd-master-0            0/1     Running             0          26m
    10. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  4/4     Terminating         0          11m
    11. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  4/4     Terminating         0          11m
    12. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  0/4     Pending             0          0s
    13. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  0/4     Init:1/3            0          1s
    14. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  0/4     Init:2/3            0          2s
    15. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  0/4     PodInitializing     0          3s
    16. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  3/4     Running             0          4s
    17. etcd-guard-ci-ln-qkgs94t-72292-9clnd-master-0            1/1     Running             0          26m
    18. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  3/4     Running             0          20s
    19. etcd-ci-ln-qkgs94t-72292-9clnd-master-0                  4/4     Running             0          20s

  6. Enter the following command to review to the values:

    1. $ oc describe -n openshift-etcd pod/<ETCD_PODNAME> | grep -e HEARTBEAT_INTERVAL -e ELECTION_TIMEOUT

    These values might not have changed from the default.