k3s etcd-snapshot

etcd-snapshot - 图1Version Gate

Available as of v1.19.1+k3s1

In this section, you’ll learn how to create backups of the K3s embedded etcd datastore, and to restore the cluster from backup.

Creating Snapshots

Snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained. To configure the snapshot interval or the number of retained snapshots, refer to the options.

The snapshot directory defaults to ${data-dir}/server/db/snapshots. The data-dir value defaults to /var/lib/rancher/k3s and can be changed by setting the --data-dir flag.

Restoring a Cluster from a Snapshot

When K3s is restored from backup, the old data directory will be moved to ${data-dir}/server/db/etcd-old/. Then K3s will attempt to restore the snapshot by creating a new data directory, then starting etcd with a new K3s cluster with one etcd member.

To restore the cluster from backup:

  • Single Server
  • High Availability

Run K3s with the --cluster-reset option, with the --cluster-reset-restore-path also given:

  1. k3s server \
  2. --cluster-reset \
  3. --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

Result: A message in the logs says that K3s can be restarted without the flags. Start k3s again and should run successfully and be restored from the specified snapshot.

In this example there are 3 servers, S1, S2, and S3. The snapshot is located on S1.

  1. On S1, start K3s with the --cluster-reset option, with the --cluster-reset-restore-path also given:

    1. k3s server \
    2. --cluster-reset \
    3. --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

    Result: A message in the logs says that K3s can be restarted without the flags.

  2. On S2 and S3, stop K3s. Then delete the data directory, /var/lib/rancher/k3s/server/db/:

    1. systemctl stop k3s
    2. rm -rf /var/lib/rancher/k3s/server/db/
  3. On S1, start K3s again:

    1. systemctl start k3s
  4. On S2 and S3, start K3s again to join the restored cluster:

    1. systemctl start k3s

Options

These options can be passed in with the command line, or in the configuration file, which may be easier to use.

OptionsDescription
—etcd-disable-snapshotsDisable automatic etcd snapshots
—etcd-snapshot-schedule-cron valueSnapshot interval time in cron spec. eg. every 5 hours 0 /5 (default: 0 /12 )
—etcd-snapshot-retention valueNumber of snapshots to retain (default: 5)
—etcd-snapshot-dir valueDirectory to save db snapshots. (Default location: ${data-dir}/db/snapshots)
—cluster-resetForget all peers and become sole member of a new cluster. This can also be set with the environment variable [$K3S_CLUSTER_RESET].
—cluster-reset-restore-path valuePath to snapshot file to be restored

S3 Compatible API Support

K3s supports writing etcd snapshots to and restoring etcd snapshots from systems with S3-compatible APIs. S3 support is available for both on-demand and scheduled snapshots.

The arguments below have been added to the server subcommand. These flags exist for the etcd-snapshot subcommand as well however the --etcd-s3 portion is removed to avoid redundancy.

OptionsDescription
—etcd-s3Enable backup to S3
—etcd-s3-endpointS3 endpoint url
—etcd-s3-endpoint-caS3 custom CA cert to connect to S3 endpoint
—etcd-s3-skip-ssl-verifyDisables S3 SSL certificate validation
—etcd-s3-access-keyS3 access key
—etcd-s3-secret-keyS3 secret key
—etcd-s3-bucketS3 bucket name
—etcd-s3-regionS3 region / bucket location (optional). defaults to us-east-1
—etcd-s3-folderS3 folder

To perform an on-demand etcd snapshot and save it to S3:

  1. k3s etcd-snapshot \
  2. --s3 \
  3. --s3-bucket=<S3-BUCKET-NAME> \
  4. --s3-access-key=<S3-ACCESS-KEY> \
  5. --s3-secret-key=<S3-SECRET-KEY>

To perform an on-demand etcd snapshot restore from S3, first make sure that K3s isn’t running. Then run the following commands:

  1. k3s server \
  2. --cluster-init \
  3. --cluster-reset \
  4. --etcd-s3 \
  5. --cluster-reset-restore-path=<SNAPSHOT-NAME> \
  6. --etcd-s3-bucket=<S3-BUCKET-NAME> \
  7. --etcd-s3-access-key=<S3-ACCESS-KEY> \
  8. --etcd-s3-secret-key=<S3-SECRET-KEY>

Etcd Snapshot and Restore Subcommands

k3s supports a set of subcommands for working with your etcd snapshots.

SubcommandDescription
deleteDelete given snapshot(s)
ls, list, lList snapshots
pruneRemove snapshots that exceed the configured retention count
saveTrigger an immediate etcd snapshot

etcd-snapshot - 图2note

The save subcommand is the same as k3s etcd-snapshot. The latter will eventually be deprecated in favor of the former.

These commands will perform as expected whether the etcd snapshots are stored locally or in an S3 compatible object store.

For additional information on the etcd snapshot subcommands, run k3s etcd-snapshot.

Delete a snapshot from S3.

  1. k3s etcd-snapshot delete \
  2. --s3 \
  3. --s3-bucket=<S3-BUCKET-NAME> \
  4. --s3-access-key=<S3-ACCESS-KEY> \
  5. --s3-secret-key=<S3-SECRET-KEY> \
  6. <SNAPSHOT-NAME>

Prune local snapshots with the default retention policy (5). The prune subcommand takes an additional flag --snapshot-retention that allows for overriding the default retention policy.

  1. k3s etcd-snapshot prune
  1. k3s etcd-snapshot prune --snapshot-retention 10