Multi-Site

New in version Jewel.

A single zone configuration typically consists of one zone group containing onezone and one or more ceph-radosgw instances where you may load-balance gatewayclient requests between the instances. In a single zone configuration, typicallymultiple gateway instances point to a single Ceph storage cluster. However, Krakensupports several multi-site configuration options for the Ceph Object Gateway:

  • Multi-zone: A more advanced configuration consists of one zone group andmultiple zones, each zone with one or more ceph-radosgw instances. Each zoneis backed by its own Ceph Storage Cluster. Multiple zones in a zone groupprovides disaster recovery for the zone group should one of the zones experiencea significant failure. In Kraken, each zone is active and may receive writeoperations. In addition to disaster recovery, multiple active zones may alsoserve as a foundation for content delivery networks.

  • Multi-zone-group: Formerly called ‘regions’, Ceph Object Gateway can alsosupport multiple zone groups, each zone group with one or more zones. Objectsstored to zones in one zone group within the same realm as another zonegroup will share a global object namespace, ensuring unique object IDs acrosszone groups and zones.

  • Multiple Realms: In Kraken, the Ceph Object Gateway supports the notionof realms, which can be a single zone group or multiple zone groups anda globally unique namespace for the realm. Multiple realms provide the abilityto support numerous configurations and namespaces.

Replicating object data between zones within a zone group looks somethinglike this:../../_images/zone-sync2.pngFor additional details on setting up a cluster, see Ceph Object Gateway forProduction.

Functional Changes from Infernalis

In Kraken, you can configure each Ceph Object Gateway towork in an active-active zone configuration, allowing for writes tonon-master zones.

The multi-site configuration is stored within a container called a“realm.” The realm stores zone groups, zones, and a time “period” withmultiple epochs for tracking changes to the configuration. In Kraken,the ceph-radosgw daemons handle the synchronization,eliminating the need for a separate synchronization agent. Additionally,the new approach to synchronization allows the Ceph Object Gateway tooperate with an “active-active” configuration instead of“active-passive”.

Requirements and Assumptions

A multi-site configuration requires at least two Ceph storage clusters,preferably given a distinct cluster name. At least two Ceph objectgateway instances, one for each Ceph storage cluster.

This guide assumes at least two Ceph storage clusters are in geographicallyseparate locations; however, the configuration can work on the samesite. This guide also assumes two Ceph object gateway servers namedrgw1 and rgw2.

Important

Running a single Ceph storage cluster is NOT recommended unless you havelow latency WAN connections.

A multi-site configuration requires a master zone group and a masterzone. Additionally, each zone group requires a master zone. Zone groupsmay have one or more secondary or non-master zones.

In this guide, the rgw1 host will serve as the master zone of themaster zone group; and, the rgw2 host will serve as the secondary zoneof the master zone group.

See Pools for instructions on creating and tuning pools for CephObject Storage.

Configuring a Master Zone

All gateways in a multi-site configuration will retrieve theirconfiguration from a ceph-radosgw daemon on a host within the masterzone group and master zone. To configure your gateways in a multi-siteconfiguration, choose a ceph-radosgw instance to configure themaster zone group and master zone.

Create a Realm

A realm contains the multi-site configuration of zone groups and zonesand also serves to enforce a globally unique namespace within the realm.

Create a new realm for the multi-site configuration by opening a commandline interface on a host identified to serve in the master zone groupand zone. Then, execute the following:

  1. # radosgw-admin realm create --rgw-realm={realm-name} [--default]

For example:

  1. # radosgw-admin realm create --rgw-realm=movies --default

If the cluster will have a single realm, specify the —default flag.If —default is specified, radosgw-admin will use this realm bydefault. If —default is not specified, adding zone-groups and zonesrequires specifying either the —rgw-realm flag or the—realm-id flag to identify the realm when adding zone groups andzones.

After creating the realm, radosgw-admin will echo back the realmconfiguration. For example:

  1. {
  2.     "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62",
  3.     "name": "movies",
  4.     "current_period": "1950b710-3e63-4c41-a19e-46a715000980",
  5.     "epoch": 1
  6. }

Note

Ceph generates a unique ID for the realm, which allows the renamingof a realm if the need arises.

Create a Master Zone Group

A realm must have at least one zone group, which will serve as themaster zone group for the realm.

Create a new master zone group for the multi-site configuration byopening a command line interface on a host identified to serve in themaster zone group and zone. Then, execute the following:

  1. # radosgw-admin zonegroup create --rgw-zonegroup={name} --endpoints={url} [--rgw-realm={realm-name}|--realm-id={realm-id}] --master --default

For example:

  1. # radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=movies --master --default

If the realm will only have a single zone group, specify the—default flag. If —default is specified, radosgw-adminwill use this zone group by default when adding new zones. If—default is not specified, adding zones will require either the—rgw-zonegroup flag or the —zonegroup-id flag to identify thezone group when adding or modifying zones.

After creating the master zone group, radosgw-admin will echo backthe zone group configuration. For example:

  1. {
  2.     "id": "f1a233f5-c354-4107-b36c-df66126475a6",
  3.     "name": "us",
  4.     "api_name": "us",
  5.     "is_master": "true",
  6.     "endpoints": [
  7.         "http:\/\/rgw1:80"
  8.     ],
  9.     "hostnames": [],
  10.     "hostnames_s3webzone": [],
  11.     "master_zone": "",
  12.     "zones": [],
  13.     "placement_targets": [],
  14.     "default_placement": "",
  15.     "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62"
  16. }

Create a Master Zone

Important

Zones must be created on a Ceph Object Gateway node that will bewithin the zone.

Create a new master zone for the multi-site configuration by opening acommand line interface on a host identified to serve in the master zonegroup and zone. Then, execute the following:

  1. # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
  2.                             --rgw-zone={zone-name} \
  3.                             --master --default \
  4.                             --endpoints={http://fqdn}[,{http://fqdn}]

For example:

  1. # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east \
  2.                             --master --default \
  3.                             --endpoints={http://fqdn}[,{http://fqdn}]

Note

The —access-key and —secret aren’t specified. Thesesettings will be added to the zone once the user is created in thenext section.

Important

The following steps assume a multi-site configuration using newlyinstalled systems that aren’t storing data yet. DO NOT DELETE thedefault zone and its pools if you are already using it to storedata, or the data will be deleted and unrecoverable.

Delete Default Zone Group and Zone

Delete the default zone if it exists. Make sure to remove it fromthe default zone group first.

  1. # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
  2. # radosgw-admin period update --commit
  3. # radosgw-admin zone rm --rgw-zone=default
  4. # radosgw-admin period update --commit
  5. # radosgw-admin zonegroup delete --rgw-zonegroup=default
  6. # radosgw-admin period update --commit

Finally, delete the default pools in your Ceph storage cluster ifthey exist.

Important

The following step assumes a multi-site configuration using newlyinstalled systems that aren’t currently storing data. DO NOT DELETEthe default zone group if you are already using it to storedata.

  1. # ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
  2. # ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
  3. # ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
  4. # ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
  5. # ceph osd pool rm default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it

Create a System User

The ceph-radosgw daemons must authenticate before pulling realm andperiod information. In the master zone, create a system user tofacilitate authentication between daemons.

  1. # radosgw-admin user create --uid="{user-name}" --display-name="{Display Name}" --system

For example:

  1. # radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system

Make a note of the access_key and secret_key, as the secondaryzones will require them to authenticate with the master zone.

Finally, add the system user to the master zone.

  1. # radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key} --secret={secret}
  2. # radosgw-admin period update --commit

Update the Period

After updating the master zone configuration, update the period.

  1. # radosgw-admin period update --commit

Note

Updating the period changes the epoch, and ensures that other zoneswill receive the updated configuration.

Update the Ceph Configuration File

Update the Ceph configuration file on master zone hosts by adding thergw_zone configuration option and the name of the master zone to theinstance entry.

  1. [client.rgw.{instance-name}]
  2. ...
  3. rgw_zone={zone-name}

For example:

  1. [client.rgw.rgw1]
  2. host = rgw1
  3. rgw frontends = "civetweb port=80"
  4. rgw_zone=us-east

Start the Gateway

On the object gateway host, start and enable the Ceph Object Gatewayservice:

  1. # systemctl start ceph-radosgw@rgw.`hostname -s`
  2. # systemctl enable ceph-radosgw@rgw.`hostname -s`

Configure Secondary Zones

Zones within a zone group replicate all data to ensure that each zonehas the same data. When creating the secondary zone, execute all of thefollowing operations on a host identified to serve the secondary zone.

Note

To add a third zone, follow the same procedures as for adding thesecondary zone. Use different zone name.

Important

You must execute metadata operations, such as user creation, on ahost within the master zone. The master zone and the secondary zonecan receive bucket operations, but the secondary zone redirectsbucket operations to the master zone. If the master zone is down,bucket operations will fail.

Pull the Realm

Using the URL path, access key and secret of the master zone in themaster zone group, pull the realm configuration to the host. To pull anon-default realm, specify the realm using the —rgw-realm or—realm-id configuration options.

  1. # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}

Note

Pulling the realm also retrieves the remote’s current periodconfiguration, and makes it the current period on this host as well.

If this realm is the default realm or the only realm, make the realm thedefault realm.

  1. # radosgw-admin realm default --rgw-realm={realm-name}

Create a Secondary Zone

Important

Zones must be created on a Ceph Object Gateway node that will bewithin the zone.

Create a secondary zone for the multi-site configuration by opening acommand line interface on a host identified to serve the secondary zone.Specify the zone group ID, the new zone name and an endpoint for thezone. DO NOT use the —master or —default flags. In Kraken,all zones run in an active-active configuration bydefault; that is, a gateway client may write data to any zone and thezone will replicate the data to all other zones within the zone group.If the secondary zone should not accept write operations, specify the—read-only flag to create an active-passive configuration betweenthe master zone and the secondary zone. Additionally, provide theaccess_key and secret_key of the generated system user stored inthe master zone of the master zone group. Execute the following:

  1. # radosgw-admin zone create --rgw-zonegroup={zone-group-name}\
  2.                             --rgw-zone={zone-name} --endpoints={url} \
  3.                             --access-key={system-key} --secret={secret}\
  4.                             --endpoints=http://{fqdn}:80 \
  5.                             [--read-only]

For example:

  1. # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west \
  2.                             --access-key={system-key} --secret={secret} \
  3.                             --endpoints=http://rgw2:80

Important

The following steps assume a multi-site configuration using newlyinstalled systems that aren’t storing data. DO NOT DELETE thedefault zone and its pools if you are already using it to storedata, or the data will be lost and unrecoverable.

Delete the default zone if needed.

  1. # radosgw-admin zone rm --rgw-zone=default

Finally, delete the default pools in your Ceph storage cluster ifneeded.

  1. # ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
  2. # ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
  3. # ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
  4. # ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
  5. # ceph osd pool rm default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it

Update the Ceph Configuration File

Update the Ceph configuration file on the secondary zone hosts by addingthe rgw_zone configuration option and the name of the secondary zoneto the instance entry.

  1. [client.rgw.{instance-name}]
  2. ...
  3. rgw_zone={zone-name}

For example:

  1. [client.rgw.rgw2]
  2. host = rgw2
  3. rgw frontends = "civetweb port=80"
  4. rgw_zone=us-west

Update the Period

After updating the master zone configuration, update the period.

  1. # radosgw-admin period update --commit

Note

Updating the period changes the epoch, and ensures that other zoneswill receive the updated configuration.

Start the Gateway

On the object gateway host, start and enable the Ceph Object Gatewayservice:

  1. # systemctl start ceph-radosgw@rgw.`hostname -s`
  2. # systemctl enable ceph-radosgw@rgw.`hostname -s`

Check Synchronization Status

Once the secondary zone is up and running, check the synchronizationstatus. Synchronization copies users and buckets created in the masterzone to the secondary zone.

  1. # radosgw-admin sync status

The output will provide the status of synchronization operations. Forexample:

  1. realm f3239bc5-e1a8-4206-a81d-e1576480804d (earth)
  2.     zonegroup c50dbb7e-d9ce-47cc-a8bb-97d9b399d388 (us)
  3.          zone 4c453b70-4a16-4ce8-8185-1893b05d346e (us-west)
  4. metadata sync syncing
  5.               full sync: 0/64 shards
  6.               metadata is caught up with master
  7.               incremental sync: 64/64 shards
  8.     data sync source: 1ee9da3e-114d-4ae3-a8a4-056e8a17f532 (us-east)
  9.                       syncing
  10.                       full sync: 0/128 shards
  11.                       incremental sync: 128/128 shards
  12.                       data is caught up with source

Note

Secondary zones accept bucket operations; however, secondary zonesredirect bucket operations to the master zone and then synchronizewith the master zone to receive the result of the bucket operations.If the master zone is down, bucket operations executed on thesecondary zone will fail, but object operations should succeed.

Maintenance

Checking the Sync Status

Information about the replication status of a zone can be queried with:

  1. $ radosgw-admin sync status
  2.         realm b3bc1c37-9c44-4b89-a03b-04c269bea5da (earth)
  3.     zonegroup f54f9b22-b4b6-4a0e-9211-fa6ac1693f49 (us)
  4.          zone adce11c9-b8ed-4a90-8bc5-3fc029ff0816 (us-2)
  5.         metadata sync syncing
  6.               full sync: 0/64 shards
  7.               incremental sync: 64/64 shards
  8.               metadata is behind on 1 shards
  9.               oldest incremental change not applied: 2017-03-22 10:20:00.0.881361s
  10.     data sync source: 341c2d81-4574-4d08-ab0f-5a2a7b168028 (us-1)
  11.                       syncing
  12.                       full sync: 0/128 shards
  13.                       incremental sync: 128/128 shards
  14.                       data is caught up with source
  15.               source: 3b5d1a3f-3f27-4e4a-8f34-6072d4bb1275 (us-3)
  16.                       syncing
  17.                       full sync: 0/128 shards
  18.                       incremental sync: 128/128 shards
  19.                       data is caught up with source

Changing the Metadata Master Zone

Important

Care must be taken when changing which zone is the metadatamaster. If a zone has not finished syncing metadata from the current masterzone, it will be unable to serve any remaining entries when promoted tomaster and those changes will be lost. For this reason, waiting for azone’s radosgw-admin sync status to catch up on metadata sync beforepromoting it to master is recommended.

Similarly, if changes to metadata are being processed by the current masterzone while another zone is being promoted to master, those changes arelikely to be lost. To avoid this, shutting down any radosgw instanceson the previous master zone is recommended. After promoting another zone,its new period can be fetched with radosgw-admin period pull and thegateway(s) can be restarted.

To promote a zone (for example, zone us-2 in zonegroup us) to metadatamaster, run the following commands on that zone:

  1. $ radosgw-admin zone modify --rgw-zone=us-2 --master
  2. $ radosgw-admin zonegroup modify --rgw-zonegroup=us --master
  3. $ radosgw-admin period update --commit

This will generate a new period, and the radosgw instance(s) in zone us-2will send this period to other zones.

Failover and Disaster Recovery

If the master zone should fail, failover to the secondary zone fordisaster recovery.

  • Make the secondary zone the master and default zone. For example:
  1. # radosgw-admin zone modify --rgw-zone={zone-name} --master --default

By default, Ceph Object Gateway will run in an active-activeconfiguration. If the cluster was configured to run in anactive-passive configuration, the secondary zone is a read-only zone.Remove the —read-only status to allow the zone to receive writeoperations. For example:

  1. # radosgw-admin zone modify --rgw-zone={zone-name} --master --default \
  2.                             --read-only=false
  • Update the period to make the changes take effect.
  1. # radosgw-admin period update --commit
  • Finally, restart the Ceph Object Gateway.
  1. # systemctl restart ceph-radosgw@rgw.`hostname -s`

If the former master zone recovers, revert the operation.

  • From the recovered zone, pull the latest realm configurationfrom the current master zone.
  1. # radosgw-admin realm pull --url={url-to-master-zone-gateway} \
  2.                            --access-key={access-key} --secret={secret}
  • Make the recovered zone the master and default zone.
  1. # radosgw-admin zone modify --rgw-zone={zone-name} --master --default
  • Update the period to make the changes take effect.
  1. # radosgw-admin period update --commit
  • Then, restart the Ceph Object Gateway in the recovered zone.
  1. # systemctl restart ceph-radosgw@rgw.`hostname -s`
  • If the secondary zone needs to be a read-only configuration, updatethe secondary zone.
  1. # radosgw-admin zone modify --rgw-zone={zone-name} --read-only
  • Update the period to make the changes take effect.
  1. # radosgw-admin period update --commit
  • Finally, restart the Ceph Object Gateway in the secondary zone.
  1. # systemctl restart ceph-radosgw@rgw.`hostname -s`

Migrating a Single Site System to Multi-Site

To migrate from a single site system with a default zone group andzone to a multi site system, use the following steps:

  • Create a realm. Replace <name> with the realm name.
  1. # radosgw-admin realm create --rgw-realm=<name> --default
  • Rename the default zone and zonegroup. Replace <name> with thezonegroup or zone name.
  1. # radosgw-admin zonegroup rename --rgw-zonegroup default --zonegroup-new-name=<name>
  2. # radosgw-admin zone rename --rgw-zone default --zone-new-name us-east-1 --rgw-zonegroup=<name>
  • Configure the master zonegroup. Replace <name> with the realm orzonegroup name. Replace <fqdn> with the fully qualified domainname(s) in the zonegroup.
  1. # radosgw-admin zonegroup modify --rgw-realm=<name> --rgw-zonegroup=<name> --endpoints http://<fqdn>:80 --master --default
  • Configure the master zone. Replace <name> with the realm,zonegroup or zone name. Replace <fqdn> with the fully qualifieddomain name(s) in the zonegroup.
  1. # radosgw-admin zone modify --rgw-realm=<name> --rgw-zonegroup=<name> \
  2.                             --rgw-zone=<name> --endpoints http://<fqdn>:80 \
  3.                             --access-key=<access-key> --secret=<secret-key> \
  4.                             --master --default
  • Create a system user. Replace <user-id> with the username.Replace <display-name> with a display name. It may containspaces.
  1. # radosgw-admin user create --uid=<user-id> --display-name="<display-name>"\
  2.                             --access-key=<access-key> --secret=<secret-key> --system
  • Commit the updated configuration.
  1. # radosgw-admin period update --commit
  • Finally, restart the Ceph Object Gateway.
  1. # systemctl restart ceph-radosgw@rgw.`hostname -s`

After completing this procedure, proceed to Configure a SecondaryZone to create a secondary zonein the master zone group.

Multi-Site Configuration Reference

The following sections provide additional details and command-lineusage for realms, periods, zone groups and zones.

Realms

A realm represents a globally unique namespace consisting of one or morezonegroups containing one or more zones, and zones containing buckets,which in turn contain objects. A realm enables the Ceph Object Gatewayto support multiple namespaces and their configuration on the samehardware.

A realm contains the notion of periods. Each period represents the stateof the zone group and zone configuration in time. Each time you make achange to a zonegroup or zone, update the period and commit it.

By default, the Ceph Object Gateway does not create a realmfor backward compatibility with Infernalis and earlier releases.However, as a best practice, we recommend creating realms for newclusters.

Create a Realm

To create a realm, execute realm create and specify the realm name.If the realm is the default, specify —default.

  1. # radosgw-admin realm create --rgw-realm={realm-name} [--default]

For example:

  1. # radosgw-admin realm create --rgw-realm=movies --default

By specifying —default, the realm will be called implicitly witheach radosgw-admin call unless —rgw-realm and the realm nameare explicitly provided.

Make a Realm the Default

One realm in the list of realms should be the default realm. There maybe only one default realm. If there is only one realm and it wasn’tspecified as the default realm when it was created, make it the defaultrealm. Alternatively, to change which realm is the default, execute:

  1. # radosgw-admin realm default --rgw-realm=movies

Note

When the realm is default, the command line assumes—rgw-realm=<realm-name> as an argument.

Delete a Realm

To delete a realm, execute realm delete and specify the realm name.

  1. # radosgw-admin realm delete --rgw-realm={realm-name}

For example:

  1. # radosgw-admin realm delete --rgw-realm=movies

Get a Realm

To get a realm, execute realm get and specify the realm name.

  1. #radosgw-admin realm get --rgw-realm=<name>

For example:

  1. # radosgw-admin realm get --rgw-realm=movies [> filename.json]

The CLI will echo a JSON object with the realm properties.

  1. {
  2.     "id": "0a68d52e-a19c-4e8e-b012-a8f831cb3ebc",
  3.     "name": "movies",
  4.     "current_period": "b0c5bbef-4337-4edd-8184-5aeab2ec413b",
  5.     "epoch": 1
  6. }

Use > and an output file name to output the JSON object to a file.

Set a Realm

To set a realm, execute realm set, specify the realm name, and—infile= with an input file name.

  1. #radosgw-admin realm set --rgw-realm=<name> --infile=<infilename>

For example:

  1. # radosgw-admin realm set --rgw-realm=movies --infile=filename.json

List Realms

To list realms, execute realm list.

  1. # radosgw-admin realm list

List Realm Periods

To list realm periods, execute realm list-periods.

  1. # radosgw-admin realm list-periods

Pull a Realm

To pull a realm from the node containing the master zone group andmaster zone to a node containing a secondary zone group or zone, executerealm pull on the node that will receive the realm configuration.

  1. # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}

Rename a Realm

A realm is not part of the period. Consequently, renaming the realm isonly applied locally, and will not get pulled with realm pull. Whenrenaming a realm with multiple zones, run the command on each zone. Torename a realm, execute the following:

  1. # radosgw-admin realm rename --rgw-realm=<current-name> --realm-new-name=<new-realm-name>

Note

DO NOT use realm set to change the name parameter. Thatchanges the internal name only. Specifying —rgw-realm wouldstill use the old realm name.

Zone Groups

The Ceph Object Gateway supports multi-site deployments and a globalnamespace by using the notion of zone groups. Formerly called a regionin Infernalis, a zone group defines the geographic location of one or more CephObject Gateway instances within one or more zones.

Configuring zone groups differs from typical configuration procedures,because not all of the settings end up in a Ceph configuration file. Youcan list zone groups, get a zone group configuration, and set a zonegroup configuration.

Create a Zone Group

Creating a zone group consists of specifying the zone group name.Creating a zone assumes it will live in the default realm unless—rgw-realm=<realm-name> is specified. If the zonegroup is thedefault zonegroup, specify the —default flag. If the zonegroup isthe master zonegroup, specify the —master flag. For example:

  1. # radosgw-admin zonegroup create --rgw-zonegroup=<name> [--rgw-realm=<name>][--master] [--default]

Note

Use zonegroup modify —rgw-zonegroup=<zonegroup-name> to modifyan existing zone group’s settings.

Make a Zone Group the Default

One zonegroup in the list of zonegroups should be the default zonegroup.There may be only one default zonegroup. If there is only one zonegroupand it wasn’t specified as the default zonegroup when it was created,make it the default zonegroup. Alternatively, to change which zonegroupis the default, execute:

  1. # radosgw-admin zonegroup default --rgw-zonegroup=comedy

Note

When the zonegroup is default, the command line assumes—rgw-zonegroup=<zonegroup-name> as an argument.

Then, update the period:

  1. # radosgw-admin period update --commit

Add a Zone to a Zone Group

To add a zone to a zonegroup, execute the following:

  1. # radosgw-admin zonegroup add --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

Remove a Zone from a Zone Group

To remove a zone from a zonegroup, execute the following:

  1. # radosgw-admin zonegroup remove --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

Rename a Zone Group

To rename a zonegroup, execute the following:

  1. # radosgw-admin zonegroup rename --rgw-zonegroup=<name> --zonegroup-new-name=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

Delete a Zone Group

To delete a zonegroup, execute the following:

  1. # radosgw-admin zonegroup delete --rgw-zonegroup=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

List Zone Groups

A Ceph cluster contains a list of zone groups. To list the zone groups,execute:

  1. # radosgw-admin zonegroup list

The radosgw-admin returns a JSON formatted list of zone groups.

  1. {
  2.     "default_info": "90b28698-e7c3-462c-a42d-4aa780d24eda",
  3.     "zonegroups": [
  4.         "us"
  5.     ]
  6. }

Get a Zone Group Map

To list the details of each zone group, execute:

  1. # radosgw-admin zonegroup-map get

Note

If you receive a failed to read zonegroup map error, runradosgw-admin zonegroup-map update as root first.

Get a Zone Group

To view the configuration of a zone group, execute:

  1. radosgw-admin zonegroup get [--rgw-zonegroup=<zonegroup>]

The zone group configuration looks like this:

  1. {
  2.     "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
  3.     "name": "us",
  4.     "api_name": "us",
  5.     "is_master": "true",
  6.     "endpoints": [
  7.         "http:\/\/rgw1:80"
  8.     ],
  9.     "hostnames": [],
  10.     "hostnames_s3website": [],
  11.     "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
  12.     "zones": [
  13.         {
  14.             "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
  15.             "name": "us-east",
  16.             "endpoints": [
  17.                 "http:\/\/rgw1"
  18.             ],
  19.             "log_meta": "true",
  20.             "log_data": "true",
  21.             "bucket_index_max_shards": 0,
  22.             "read_only": "false"
  23.         },
  24.         {
  25.             "id": "d1024e59-7d28-49d1-8222-af101965a939",
  26.             "name": "us-west",
  27.             "endpoints": [
  28.                 "http:\/\/rgw2:80"
  29.             ],
  30.             "log_meta": "false",
  31.             "log_data": "true",
  32.             "bucket_index_max_shards": 0,
  33.             "read_only": "false"
  34.         }
  35.     ],
  36.     "placement_targets": [
  37.         {
  38.             "name": "default-placement",
  39.             "tags": []
  40.         }
  41.     ],
  42.     "default_placement": "default-placement",
  43.     "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
  44. }

Set a Zone Group

Defining a zone group consists of creating a JSON object, specifying atleast the required settings:

  • name: The name of the zone group. Required.

  • api_name: The API name for the zone group. Optional.

  • is_master: Determines if the zone group is the master zone group.Required. note: You can only have one master zone group.

  • endpoints: A list of all the endpoints in the zone group. Forexample, you may use multiple domain names to refer to the same zonegroup. Remember to escape the forward slashes (\/). You may alsospecify a port (fqdn:port) for each endpoint. Optional.

  • hostnames: A list of all the hostnames in the zone group. Forexample, you may use multiple domain names to refer to the same zonegroup. Optional. The rgw dns name setting will automatically beincluded in this list. You should restart the gateway daemon(s) afterchanging this setting.

  • master_zone: The master zone for the zone group. Optional. Usesthe default zone if not specified. note: You can only have onemaster zone per zone group.

  • zones: A list of all zones within the zone group. Each zone has aname (required), a list of endpoints (optional), and whether or notthe gateway will log metadata and data operations (false by default).

  • placement_targets: A list of placement targets (optional). Eachplacement target contains a name (required) for the placement targetand a list of tags (optional) so that only users with the tag can usethe placement target (i.e., the user’s placement_tags field inthe user info).

  • default_placement: The default placement target for the objectindex and object data. Set to default-placement by default. Youmay also set a per-user default placement in the user info for eachuser.

To set a zone group, create a JSON object consisting of the requiredfields, save the object to a file (e.g., zonegroup.json); then,execute the following command:

  1. # radosgw-admin zonegroup set --infile zonegroup.json

Where zonegroup.json is the JSON file you created.

Important

The default zone group is_master setting is true bydefault. If you create a new zone group and want to make it themaster zone group, you must either set the default zone groupis_master setting to false, or delete the default zonegroup.

Finally, update the period:

  1. # radosgw-admin period update --commit

Set a Zone Group Map

Setting a zone group map consists of creating a JSON object consistingof one or more zone groups, and setting the master_zonegroup for thecluster. Each zone group in the zone group map consists of a key/valuepair, where the key setting is equivalent to the name settingfor an individual zone group configuration, and the val is a JSONobject consisting of an individual zone group configuration.

You may only have one zone group with is_master equal to true,and it must be specified as the master_zonegroup at the end of thezone group map. The following JSON object is an example of a defaultzone group map.

  1. {
  2.     "zonegroups": [
  3.         {
  4.             "key": "90b28698-e7c3-462c-a42d-4aa780d24eda",
  5.             "val": {
  6.                 "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
  7.                 "name": "us",
  8.                 "api_name": "us",
  9.                 "is_master": "true",
  10.                 "endpoints": [
  11.                     "http:\/\/rgw1:80"
  12.                 ],
  13.                 "hostnames": [],
  14.                 "hostnames_s3website": [],
  15.                 "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
  16.                 "zones": [
  17.                     {
  18.                         "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
  19.                         "name": "us-east",
  20.                         "endpoints": [
  21.                             "http:\/\/rgw1"
  22.                         ],
  23.                         "log_meta": "true",
  24.                         "log_data": "true",
  25.                         "bucket_index_max_shards": 0,
  26.                         "read_only": "false"
  27.                     },
  28.                     {
  29.                         "id": "d1024e59-7d28-49d1-8222-af101965a939",
  30.                         "name": "us-west",
  31.                         "endpoints": [
  32.                             "http:\/\/rgw2:80"
  33.                         ],
  34.                         "log_meta": "false",
  35.                         "log_data": "true",
  36.                         "bucket_index_max_shards": 0,
  37.                         "read_only": "false"
  38.                     }
  39.                 ],
  40.                 "placement_targets": [
  41.                     {
  42.                         "name": "default-placement",
  43.                         "tags": []
  44.                     }
  45.                 ],
  46.                 "default_placement": "default-placement",
  47.                 "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
  48.             }
  49.         }
  50.     ],
  51.     "master_zonegroup": "90b28698-e7c3-462c-a42d-4aa780d24eda",
  52.     "bucket_quota": {
  53.         "enabled": false,
  54.         "max_size_kb": -1,
  55.         "max_objects": -1
  56.     },
  57.     "user_quota": {
  58.         "enabled": false,
  59.         "max_size_kb": -1,
  60.         "max_objects": -1
  61.     }
  62. }

To set a zone group map, execute the following:

  1. # radosgw-admin zonegroup-map set --infile zonegroupmap.json

Where zonegroupmap.json is the JSON file you created. Ensure thatyou have zones created for the ones specified in the zone group map.Finally, update the period.

  1. # radosgw-admin period update --commit

Zones

Ceph Object Gateway supports the notion of zones. A zone defines alogical group consisting of one or more Ceph Object Gateway instances.

Configuring zones differs from typical configuration procedures, becausenot all of the settings end up in a Ceph configuration file. You canlist zones, get a zone configuration and set a zone configuration.

Create a Zone

To create a zone, specify a zone name. If it is a master zone, specifythe —master option. Only one zone in a zone group may be a masterzone. To add the zone to a zonegroup, specify the —rgw-zonegroupoption with the zonegroup name.

  1. # radosgw-admin zone create --rgw-zone=<name> \
  2.                 [--zonegroup=<zonegroup-name]\
  3.                 [--endpoints=<endpoint>[,<endpoint>] \
  4.                 [--master] [--default] \
  5.                 --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY

Then, update the period:

  1. # radosgw-admin period update --commit

Delete a Zone

To delete zone, first remove it from the zonegroup.

  1. # radosgw-admin zonegroup remove --zonegroup=<name>\
  2.                                  --zone=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

Next, delete the zone. Execute the following:

  1. # radosgw-admin zone rm --rgw-zone<name>

Finally, update the period:

  1. # radosgw-admin period update --commit

Important

Do not delete a zone without removing it from a zone group first.Otherwise, updating the period will fail.

If the pools for the deleted zone will not be used anywhere else,consider deleting the pools. Replace <del-zone> in the example belowwith the deleted zone’s name.

Important

Only delete the pools with prepended zone names. Deleting the rootpool, such as, .rgw.root will remove all of the system’sconfiguration.

Important

Once the pools are deleted, all of the data within them are deletedin an unrecoverable manner. Only delete the pools if the poolcontents are no longer needed.

  1. # ceph osd pool rm <del-zone>.rgw.control <del-zone>.rgw.control --yes-i-really-really-mean-it
  2. # ceph osd pool rm <del-zone>.rgw.data.root <del-zone>.rgw.data.root --yes-i-really-really-mean-it
  3. # ceph osd pool rm <del-zone>.rgw.gc <del-zone>.rgw.gc --yes-i-really-really-mean-it
  4. # ceph osd pool rm <del-zone>.rgw.log <del-zone>.rgw.log --yes-i-really-really-mean-it
  5. # ceph osd pool rm <del-zone>.rgw.users.uid <del-zone>.rgw.users.uid --yes-i-really-really-mean-it

Modify a Zone

To modify a zone, specify the zone name and the parameters you wish tomodify.

  1. # radosgw-admin zone modify [options]

Where [options]:

  • —access-key=<key>

  • —secret/—secret-key=<key>

  • —master

  • —default

  • —endpoints=<list>

Then, update the period:

  1. # radosgw-admin period update --commit

List Zones

As root, to list the zones in a cluster, execute:

  1. # radosgw-admin zone list

Get a Zone

As root, to get the configuration of a zone, execute:

  1. # radosgw-admin zone get [--rgw-zone=<zone>]

The default zone looks like this:

  1. { "domain_root": ".rgw",
  2.   "control_pool": ".rgw.control",
  3.   "gc_pool": ".rgw.gc",
  4.   "log_pool": ".log",
  5.   "intent_log_pool": ".intent-log",
  6.   "usage_log_pool": ".usage",
  7.   "user_keys_pool": ".users",
  8.   "user_email_pool": ".users.email",
  9.   "user_swift_pool": ".users.swift",
  10.   "user_uid_pool": ".users.uid",
  11.   "system_key": { "access_key": "", "secret_key": ""},
  12.   "placement_pools": [
  13.       {  "key": "default-placement",
  14.          "val": { "index_pool": ".rgw.buckets.index",
  15.                   "data_pool": ".rgw.buckets"}
  16.       }
  17.     ]
  18.   }

Set a Zone

Configuring a zone involves specifying a series of Ceph Object Gatewaypools. For consistency, we recommend using a pool prefix that is thesame as the zone name. SeePoolsfor details of configuring pools.

To set a zone, create a JSON object consisting of the pools, save theobject to a file (e.g., zone.json); then, execute the followingcommand, replacing {zone-name} with the name of the zone:

  1. # radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json

Where zone.json is the JSON file you created.

Then, as root, update the period:

  1. # radosgw-admin period update --commit

Rename a Zone

To rename a zone, specify the zone name and the new zone name.

  1. # radosgw-admin zone rename --rgw-zone=<name> --zone-new-name=<name>

Then, update the period:

  1. # radosgw-admin period update --commit

Zone Group and Zone Settings

When configuring a default zone group and zone, the pool name includesthe zone name. For example:

  • default.rgw.control

To change the defaults, include the following settings in your Cephconfiguration file under each [client.radosgw.{instance-name}]instance.

NameDescriptionTypeDefault
rgw_zoneThe name of the zone for thegateway instance.StringNone
rgw_zonegroupThe name of the zone group forthe gateway instance.StringNone
rgw_zonegroup_root_poolThe root pool for the zone group.String.rgw.root
rgw_zone_root_poolThe root pool for the zone.String.rgw.root
rgw_default_zone_group_info_oidThe OID for storing the defaultzone group. We do not recommendchanging this setting.Stringdefault.zonegroup