RBD Mirroring

RBD images can be asynchronously mirrored between two Ceph clusters. Thiscapability uses the RBD journaling image feature to ensure crash-consistentreplication between clusters. Mirroring is configured on a per-pool basiswithin peer clusters and can be configured to automatically mirror allimages within a pool or only a specific subset of images. Mirroring isconfigured using the rbd command. The rbd-mirror daemon is responsiblefor pulling image updates from the remote, peer cluster and applying them tothe image within the local cluster.

Note

RBD mirroring requires the Ceph Jewel release or later.

Depending on the desired needs for replication, RBD mirroring can be configuredfor either one- or two-way replication:

  • One-way Replication: When data is only mirrored from a primary cluster toa secondary cluster, the rbd-mirror daemon runs only on the secondarycluster.

  • Two-way Replication: When data is mirrored from primary images on onecluster to non-primary images on another cluster (and vice-versa), therbd-mirror daemon runs on both clusters.

Important

Each instance of the rbd-mirror daemon must be able toconnect to both the local and remote Ceph clusters simultaneously (i.e.all monitor and OSD hosts). Additionally, the network must have sufficientbandwidth between the two data centers to handle mirroring workload.

Pool Configuration

The following procedures demonstrate how to perform the basic administrativetasks to configure mirroring using the rbd command. Mirroring isconfigured on a per-pool basis within the Ceph clusters.

The pool configuration steps should be performed on both peer clusters. Theseprocedures assume two clusters, named “site-a” and “site-b”, are accessible froma single host for clarity.

See the rbd manpage for additional details of how to connect to differentCeph clusters.

Note

The cluster name in the following examples corresponds to a Cephconfiguration file of the same name (e.g. /etc/ceph/site-b.conf). See theceph-conf documentation for how to configure multiple clusters.

Enable Mirroring

To enable mirroring on a pool with rbd, specify the mirror pool enablecommand, the pool name, and the mirroring mode:

  1. rbd mirror pool enable {pool-name} {mode}

The mirroring mode can either be pool or image:

  • pool: When configured in pool mode, all images in the pool with thejournaling feature enabled are mirrored.

  • image: When configured in image mode, mirroring needs to beexplicitly enabled on each image.

For example:

  1. $ rbd --cluster site-a mirror pool enable image-pool pool
  2. $ rbd --cluster site-b mirror pool enable image-pool pool

Disable Mirroring

To disable mirroring on a pool with rbd, specify the mirror pool disablecommand and the pool name:

  1. rbd mirror pool disable {pool-name}

When mirroring is disabled on a pool in this way, mirroring will also bedisabled on any images (within the pool) for which mirroring was enabledexplicitly.

For example:

  1. $ rbd --cluster site-a mirror pool disable image-pool
  2. $ rbd --cluster site-b mirror pool disable image-pool

Bootstrap Peers

In order for the rbd-mirror daemon to discover its peer cluster, the peerneeds to be registered to the pool and a user account needs to be created.This process can be automated with rbd and themirror pool peer bootstrap create and mirror pool peer bootstrap importcommands.

To manually create a new bootstrap token with rbd, specify themirror pool peer bootstrap create command, a pool name, along with anoptional friendly site name to describe the local cluster:

  1. rbd mirror pool peer bootstrap create [--site-name {local-site-name}] {pool-name}

The output of mirror pool peer bootstrap create will be a token that shouldbe provided to the mirror pool peer bootstrap import command. For example,on site-a:

  1. $ rbd --cluster site-a mirror pool peer bootstrap create --site-name site-a image-pool
  2. eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==

To manually import the bootstrap token created by another cluster with rbd,specify the mirror pool peer bootstrap import command, the pool name, a filepath to the created token (or ‘-‘ to read from standard input), along with anoptional friendly site name to describe the local cluster and a mirroringdirection (defaults to rx-tx for bidirectional mirroring, but can also be setto rx-only for unidirectional mirroring):

  1. rbd mirror pool peer bootstrap import [--site-name {local-site-name}] [--direction {rx-only or rx-tx}] {pool-name} {token-path}

For example, on site-b:

  1. $ cat <<EOF > token
  2. eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==
  3. EOF
  4. $ rbd --cluster site-b mirror pool peer bootstrap import --site-name site-b image-pool token

Add Cluster Peer Manually

Cluster peers can be specified manually if desired or if the above bootstrapcommands are not available with the currently installed Ceph release.

The remote rbd-mirror daemon will need access to the local cluster toperform mirroring. A new local Ceph user should be created for the remotedaemon to use. To create a Ceph user, with ceph specify theauth get-or-create command, user name, monitor caps, and OSD caps:

  1. ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd' osd 'profile rbd'

The resulting keyring should be copied to the other cluster’s rbd-mirrordaemon hosts if not using the Ceph monitor config-key store described below.

To manually add a mirroring peer Ceph cluster with rbd, specify themirror pool peer add command, the pool name, and a cluster specification:

  1. rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}

For example:

  1. $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b
  2. $ rbd --cluster site-b mirror pool peer add image-pool client.rbd-mirror-peer@site-a

By default, the rbd-mirror daemon needs to have access to a Cephconfiguration file located at /etc/ceph/{cluster-name}.conf that providesthe addresses of the peer cluster’s monitors, in addition to a keyring for{client-name} located in the default or configured keyring search paths(e.g. /etc/ceph/{cluster-name}.{client-name}.keyring).

Alternatively, the peer cluster’s monitor and/or client key can be securelystored within the local Ceph monitor config-key store. To specify thepeer cluster connection attributes when adding a mirroring peer, use the—remote-mon-host and —remote-key-file optionals. For example:

  1. $ cat <<EOF > remote-key-file
  2. AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==
  3. EOF
  4. $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b --remote-mon-host 192.168.1.1,192.168.1.2 --remote-key-file remote-key-file
  5. $ rbd --cluster site-a mirror pool info image-pool --all
  6. Mode: pool
  7. Peers:
  8.   UUID                                 NAME   CLIENT                 MON_HOST                KEY
  9.   587b08db-3d33-4f32-8af8-421e77abb081 site-b client.rbd-mirror-peer 192.168.1.1,192.168.1.2 AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==

Remove Cluster Peer

To remove a mirroring peer Ceph cluster with rbd, specify themirror pool peer remove command, the pool name, and the peer UUID(available from the rbd mirror pool info command):

  1. rbd mirror pool peer remove {pool-name} {peer-uuid}

For example:

  1. $ rbd --cluster site-a mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445
  2. $ rbd --cluster site-b mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08

Data Pools

When creating images in the destination cluster, rbd-mirror selects a datapool as follows:

  • If the destination cluster has a default data pool configured (with therbd_default_data_pool configuration option), it will be used.

  • Otherwise, if the source image uses a separate data pool, and a pool with thesame name exists on the destination cluster, that pool will be used.

  • If neither of the above is true, no data pool will be set.

Image Configuration

Unlike pool configuration, image configuration only needs to be performed againsta single mirroring peer Ceph cluster.

Mirrored RBD images are designated as either primary or non-primary. This is aproperty of the image and not the pool. Images that are designated asnon-primary cannot be modified.

Images are automatically promoted to primary when mirroring is first enabled onan image (either implicitly if the pool mirror mode was pool and the imagehas the journaling image feature enabled, or explicitly enabled by therbd command).

Enable Image Journaling Support

RBD mirroring uses the RBD journaling feature to ensure that the replicatedimage always remains crash-consistent. Before an image can be mirrored toa peer cluster, the journaling feature must be enabled. The feature can beenabled at image creation time by providing the—image-feature exclusive-lock,journaling option to the rbd command.

Alternatively, the journaling feature can be dynamically enabled onpre-existing RBD images. To enable journaling with rbd, specifythe feature enable command, the pool and image name, and the feature name:

  1. rbd feature enable {pool-name}/{image-name} {feature-name}

For example:

  1. $ rbd --cluster site-a feature enable image-pool/image-1 journaling

Note

The journaling feature is dependent on the exclusive-lock feature. Ifthe exclusive-lock feature is not already enabled, it should be enabled priorto enabling the journaling feature.

Tip

You can enable journaling on all new images by default by addingrbd default features = 125 to your Ceph configuration file.

Enable Image Mirroring

If the mirroring is configured in image mode for the image’s pool, then itis necessary to explicitly enable mirroring for each image within the pool.To enable mirroring for a specific image with rbd, specify themirror image enable command along with the pool and image name:

  1. rbd mirror image enable {pool-name}/{image-name}

For example:

  1. $ rbd --cluster site-a mirror image enable image-pool/image-1

Disable Image Mirroring

To disable mirroring for a specific image with rbd, specify themirror image disable command along with the pool and image name:

  1. rbd mirror image disable {pool-name}/{image-name}

For example:

  1. $ rbd --cluster site-a mirror image disable image-pool/image-1

Image Promotion and Demotion

In a failover scenario where the primary designation needs to be moved to theimage in the peer Ceph cluster, access to the primary image should be stopped(e.g. power down the VM or remove the associated drive from a VM), demote thecurrent primary image, promote the new primary image, and resume access to theimage on the alternate cluster.

Note

RBD only provides the necessary tools to facilitate an orderlyfailover of an image. An external mechanism is required to coordinate thefull failover process (e.g. closing the image before demotion).

To demote a specific image to non-primary with rbd, specify themirror image demote command along with the pool and image name:

  1. rbd mirror image demote {pool-name}/{image-name}

For example:

  1. $ rbd --cluster site-a mirror image demote image-pool/image-1

To demote all primary images within a pool to non-primary with rbd, specifythe mirror pool demote command along with the pool name:

  1. rbd mirror pool demote {pool-name}

For example:

  1. $ rbd --cluster site-a mirror pool demote image-pool

To promote a specific image to primary with rbd, specify themirror image promote command along with the pool and image name:

  1. rbd mirror image promote [--force] {pool-name}/{image-name}

For example:

  1. $ rbd --cluster site-b mirror image promote image-pool/image-1

To promote all non-primary images within a pool to primary with rbd, specifythe mirror pool promote command along with the pool name:

  1. rbd mirror pool promote [--force] {pool-name}

For example:

  1. $ rbd --cluster site-a mirror pool promote image-pool

Tip

Since the primary / non-primary status is per-image, it is possible tohave two clusters split the IO load and stage failover / failback.

Note

Promotion can be forced using the —force option. Forcedpromotion is needed when the demotion cannot be propagated to the peerCeph cluster (e.g. Ceph cluster failure, communication outage). This willresult in a split-brain scenario between the two peers and the image will nolonger be in-sync until a force resync command is issued.

Force Image Resync

If a split-brain event is detected by the rbd-mirror daemon, it will notattempt to mirror the affected image until corrected. To resume mirroring for animage, first demote the image determined to be out-of-date and then request aresync to the primary image. To request an image resync with rbd, specify themirror image resync command along with the pool and image name:

  1. rbd mirror image resync {pool-name}/{image-name}

For example:

  1. $ rbd mirror image resync image-pool/image-1

Note

The rbd command only flags the image as requiring a resync. Thelocal cluster’s rbd-mirror daemon process is responsible for performingthe resync asynchronously.

Mirror Status

The peer cluster replication status is stored for every primary mirrored image.This status can be retrieved using the mirror image status andmirror pool status commands.

To request the mirror image status with rbd, specify themirror image status command along with the pool and image name:

  1. rbd mirror image status {pool-name}/{image-name}

For example:

  1. $ rbd mirror image status image-pool/image-1

To request the mirror pool summary status with rbd, specify themirror pool status command along with the pool name:

  1. rbd mirror pool status {pool-name}

For example:

  1. $ rbd mirror pool status image-pool

Note

Adding —verbose option to the mirror pool status command willadditionally output status details for every mirroring image in the pool.

rbd-mirror Daemon

The two rbd-mirror daemons are responsible for watching image journals on theremote, peer cluster and replaying the journal events against the localcluster. The RBD image journaling feature records all modifications to theimage in the order they occur. This ensures that a crash-consistent mirror ofthe remote image is available locally.

The rbd-mirror daemon is available within the optional rbd-mirrordistribution package.

Important

Each rbd-mirror daemon requires the ability to connectto both clusters simultaneously.

Warning

Pre-Luminous releases: only run a single rbd-mirror daemon perCeph cluster.

Each rbd-mirror daemon should use a unique Ceph user ID. Tocreate a Ceph user, with ceph specify the auth get-or-createcommand, user name, monitor caps, and OSD caps:

  1. ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd-mirror' osd 'profile rbd'

The rbd-mirror daemon can be managed by systemd by specifying the userID as the daemon instance:

  1. systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id}

The rbd-mirror can also be run in foreground by rbd-mirror command:

  1. rbd-mirror -f --log-file={log_path}