Manual Deployment

All Ceph clusters require at least one monitor, and at least as many OSDs ascopies of an object stored on the cluster. Bootstrapping the initial monitor(s)is the first step in deploying a Ceph Storage Cluster. Monitor deployment alsosets important criteria for the entire cluster, such as the number of replicasfor pools, the number of placement groups per OSD, the heartbeat intervals,whether authentication is required, etc. Most of these values are set bydefault, so it’s useful to know about them when setting up your cluster forproduction.

Following the same configuration as Installation (Quick), we will set up acluster with node1 as the monitor node, and node2 and node3 forOSD nodes.

Manual Deployment - 图1

Monitor Bootstrapping

Bootstrapping a monitor (a Ceph Storage Cluster, in theory) requiresa number of things:

  • Unique Identifier: The fsid is a unique identifier for the cluster,and stands for File System ID from the days when the Ceph Storage Cluster wasprincipally for the Ceph File System. Ceph now supports native interfaces,block devices, and object storage gateway interfaces too, so fsid is abit of a misnomer.

  • Cluster Name: Ceph clusters have a cluster name, which is a simple stringwithout spaces. The default cluster name is ceph, but you may specifya different cluster name. Overriding the default cluster name isespecially useful when you are working with multiple clusters and you need toclearly understand which cluster your are working with.

For example, when you run multiple clusters in a multisite configuration,the cluster name (e.g., us-west, us-east) identifies the cluster forthe current CLI session. Note: To identify the cluster name on thecommand line interface, specify the Ceph configuration file with thecluster name (e.g., ceph.conf, us-west.conf, us-east.conf, etc.).Also see CLI usage (ceph —cluster {cluster-name}).

  • Monitor Name: Each monitor instance within a cluster has a unique name.In common practice, the Ceph Monitor name is the host name (we recommend oneCeph Monitor per host, and no commingling of Ceph OSD Daemons withCeph Monitors). You may retrieve the short hostname with hostname -s.

  • Monitor Map: Bootstrapping the initial monitor(s) requires you togenerate a monitor map. The monitor map requires the fsid, the clustername (or uses the default), and at least one host name and its IP address.

  • Monitor Keyring: Monitors communicate with each other via asecret key. You must generate a keyring with a monitor secret and provideit when bootstrapping the initial monitor(s).

  • Administrator Keyring: To use the ceph CLI tools, you must havea client.admin user. So you must generate the admin user and keyring,and you must also add the client.admin user to the monitor keyring.

The foregoing requirements do not imply the creation of a Ceph Configurationfile. However, as a best practice, we recommend creating a Ceph configurationfile and populating it with the fsid, the mon initial members and themon host settings.

You can get and set all of the monitor settings at runtime as well. However,a Ceph Configuration file may contain only those settings that override thedefault values. When you add settings to a Ceph configuration file, thesesettings override the default settings. Maintaining those settings in aCeph configuration file makes it easier to maintain your cluster.

The procedure is as follows:

  • Log in to the initial monitor node(s):
  1. ssh {hostname}

For example:

  1. ssh node1
  • Ensure you have a directory for the Ceph configuration file. By default,Ceph uses /etc/ceph. When you install ceph, the installer willcreate the /etc/ceph directory automatically.
  1. ls /etc/ceph

Note: Deployment tools may remove this directory when purging acluster (e.g., ceph-deploy purgedata {node-name}, ceph-deploy purge{node-name}).

  • Create a Ceph configuration file. By default, Ceph usesceph.conf, where ceph reflects the cluster name.
  1. sudo vim /etc/ceph/ceph.conf
  • Generate a unique ID (i.e., fsid) for your cluster.
  1. uuidgen
  • Add the unique ID to your Ceph configuration file.
  1. fsid = {UUID}

For example:

  1. fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
  • Add the initial monitor(s) to your Ceph configuration file.
  1. mon initial members = {hostname}[,{hostname}]

For example:

  1. mon initial members = node1
  • Add the IP address(es) of the initial monitor(s) to your Ceph configurationfile and save the file.
  1. mon host = {ip-address}[,{ip-address}]

For example:

  1. mon host = 192.168.0.1

Note: You may use IPv6 addresses instead of IPv4 addresses, butyou must set ms bind ipv6 to true. See Network ConfigurationReference for details about network configuration.

  • Create a keyring for your cluster and generate a monitor secret key.
  1. ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
  • Generate an administrator keyring, generate a client.admin user and addthe user to the keyring.
  1. sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
  • Generate a bootstrap-osd keyring, generate a client.bootstrap-osd user and addthe user to the keyring.
  1. sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
  • Add the generated keys to the ceph.mon.keyring.
  1. sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
  2. sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
  • Change the owner for ceph.mon.keyring.
  1. sudo chown ceph:ceph /tmp/ceph.mon.keyring
  • Generate a monitor map using the hostname(s), host IP address(es) and the FSID.Save it as /tmp/monmap:
  1. monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap

For example:

  1. monmaptool --create --add node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
  • Create a default data directory (or directories) on the monitor host(s).
  1. sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}

For example:

  1. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node1

See Monitor Config Reference - Data for details.

  • Populate the monitor daemon(s) with the monitor map and keyring.
  1. sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

For example:

  1. sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
  • Consider settings for a Ceph configuration file. Common settings includethe following:
  1. [global]
  2. fsid = {cluster-id}
  3. mon initial members = {hostname}[, {hostname}]
  4. mon host = {ip-address}[, {ip-address}]
  5. public network = {network}[, {network}]
  6. cluster network = {network}[, {network}]
  7. auth cluster required = cephx
  8. auth service required = cephx
  9. auth client required = cephx
  10. osd journal size = {n}
  11. osd pool default size = {n}  # Write an object n times.
  12. osd pool default min size = {n} # Allow writing n copies in a degraded state.
  13. osd pool default pg num = {n}
  14. osd pool default pgp num = {n}
  15. osd crush chooseleaf type = {n}

In the foregoing example, the [global] section of the configuration mightlook like this:

  1. [global]
  2. fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
  3. mon initial members = node1
  4. mon host = 192.168.0.1
  5. public network = 192.168.0.0/24
  6. auth cluster required = cephx
  7. auth service required = cephx
  8. auth client required = cephx
  9. osd journal size = 1024
  10. osd pool default size = 3
  11. osd pool default min size = 2
  12. osd pool default pg num = 333
  13. osd pool default pgp num = 333
  14. osd crush chooseleaf type = 1
  • Start the monitor(s).

For most distributions, services are started via systemd now:

  1. sudo systemctl start ceph-mon@node1

For older Debian/CentOS/RHEL, use sysvinit:

  1. sudo /etc/init.d/ceph start mon.node1
  • Verify that the monitor is running.
  1. ceph -s

You should see output that the monitor you started is up and running, andyou should see a health error indicating that placement groups are stuckinactive. It should look something like this:

  1. cluster:
  2.   id:     a7f64266-0894-4f1e-a635-d0aeaca0e993
  3.   health: HEALTH_OK
  4.  
  5. services:
  6.   mon: 1 daemons, quorum node1
  7.   mgr: node1(active)
  8.   osd: 0 osds: 0 up, 0 in
  9.  
  10. data:
  11.   pools:   0 pools, 0 pgs
  12.   objects: 0 objects, 0 bytes
  13.   usage:   0 kB used, 0 kB / 0 kB avail
  14.   pgs:

Note: Once you add OSDs and start them, the placement group health errorsshould disappear. See Adding OSDs for details.

Manager daemon configuration

On each node where you run a ceph-mon daemon, you should also set up a ceph-mgr daemon.

See ceph-mgr administrator’s guide

Adding OSDs

Once you have your initial monitor(s) running, you should add OSDs. Your clustercannot reach an active + clean state until you have enough OSDs to handle thenumber of copies of an object (e.g., osd pool default size = 2 requires atleast two OSDs). After bootstrapping your monitor, your cluster has a defaultCRUSH map; however, the CRUSH map doesn’t have any Ceph OSD Daemons mapped toa Ceph Node.

Short Form

Ceph provides the ceph-volume utility, which can prepare a logical volume, disk, or partitionfor use with Ceph. The ceph-volume utility creates the OSD ID byincrementing the index. Additionally, ceph-volume will add the new OSD to theCRUSH map under the host for you. Execute ceph-volume -h for CLI details.The ceph-volume utility automates the steps of the Long Form below. Tocreate the first two OSDs with the short form procedure, execute the followingon node2 and node3:

bluestore

  • Create the OSD.
  1. ssh {node-name}
  2. sudo ceph-volume lvm create --data {data-path}

For example:

  1. ssh node1
  2. sudo ceph-volume lvm create --data /dev/hdd1

Alternatively, the creation process can be split in two phases (prepare, andactivate):

  • Prepare the OSD.
  1. ssh {node-name}
  2. sudo ceph-volume lvm prepare --data {data-path} {data-path}

For example:

  1. ssh node1
  2. sudo ceph-volume lvm prepare --data /dev/hdd1

Once prepared, the ID and FSID of the prepared OSD are required foractivation. These can be obtained by listing OSDs in the current server:

  1. sudo ceph-volume lvm list
  • Activate the OSD:
  1. sudo ceph-volume lvm activate {ID} {FSID}

For example:

  1. sudo ceph-volume lvm activate 0 a7f64266-0894-4f1e-a635-d0aeaca0e993

filestore

  • Create the OSD.
  1. ssh {node-name}
  2. sudo ceph-volume lvm create --filestore --data {data-path} --journal {journal-path}

For example:

  1. ssh node1
  2. sudo ceph-volume lvm create --filestore --data /dev/hdd1 --journal /dev/hdd2

Alternatively, the creation process can be split in two phases (prepare, andactivate):

  • Prepare the OSD.
  1. ssh {node-name}
  2. sudo ceph-volume lvm prepare --filestore --data {data-path} --journal {journal-path}

For example:

  1. ssh node1
  2. sudo ceph-volume lvm prepare --filestore --data /dev/hdd1 --journal /dev/hdd2

Once prepared, the ID and FSID of the prepared OSD are required foractivation. These can be obtained by listing OSDs in the current server:

  1. sudo ceph-volume lvm list
  • Activate the OSD:
  1. sudo ceph-volume lvm activate --filestore {ID} {FSID}

For example:

  1. sudo ceph-volume lvm activate --filestore 0 a7f64266-0894-4f1e-a635-d0aeaca0e993

Long Form

Without the benefit of any helper utilities, create an OSD and add it to thecluster and CRUSH map with the following procedure. To create the first twoOSDs with the long form procedure, execute the following steps for each OSD.

Note

This procedure does not describe deployment on top of dm-cryptmaking use of the dm-crypt ‘lockbox’.

  • Connect to the OSD host and become root.
  1. ssh {node-name}
  2. sudo bash
  • Generate a UUID for the OSD.
  1. UUID=$(uuidgen)
  • Generate a cephx key for the OSD.
  1. OSD_SECRET=$(ceph-authtool --gen-print-key)
  • Create the OSD. Note that an OSD ID can be provided as anadditional argument to ceph osd new if you need to reuse apreviously-destroyed OSD id. We assume that theclient.bootstrap-osd key is present on the machine. You mayalternatively execute this command as client.admin on adifferent host where that key is present.:
  1. ID=$(echo "{\"cephx_secret\": \"$OSD_SECRET\"}" | \
  2.    ceph osd new $UUID -i - \
  3.    -n client.bootstrap-osd -k /var/lib/ceph/bootstrap-osd/ceph.keyring)

It is also possible to include a crush_device_class property in the JSONto set an initial class other than the default (ssd or hdd based onthe auto-detected device type).

  • Create the default directory on your new OSD.
  1. mkdir /var/lib/ceph/osd/ceph-$ID
  • If the OSD is for a drive other than the OS drive, prepare itfor use with Ceph, and mount it to the directory you just created.
  1. mkfs.xfs /dev/{DEV}
  2. mount /dev/{DEV} /var/lib/ceph/osd/ceph-$ID
  • Write the secret to the OSD keyring file.
  1. ceph-authtool --create-keyring /var/lib/ceph/osd/ceph-$ID/keyring \
  2.      --name osd.$ID --add-key $OSD_SECRET
  • Initialize the OSD data directory.
  1. ceph-osd -i $ID --mkfs --osd-uuid $UUID
  • Fix ownership.
  1. chown -R ceph:ceph /var/lib/ceph/osd/ceph-$ID
  • After you add an OSD to Ceph, the OSD is in your configuration. However,it is not yet running. You must startyour new OSD before it can begin receiving data.

For modern systemd distributions:

  1. systemctl enable ceph-osd@$ID
  2. systemctl start ceph-osd@$ID

For example:

  1. systemctl enable ceph-osd@12
  2. systemctl start ceph-osd@12

Adding MDS

In the below instructions, {id} is an arbitrary name, such as the hostname of the machine.

  • Create the mds data directory.:
  1. mkdir -p /var/lib/ceph/mds/{cluster-name}-{id}
  • Create a keyring.:
  1. ceph-authtool --create-keyring /var/lib/ceph/mds/{cluster-name}-{id}/keyring --gen-key -n mds.{id}
  • Import the keyring and set caps.:
  1. ceph auth add mds.{id} osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/{cluster}-{id}/keyring
  • Add to ceph.conf.:
  1. [mds.{id}]
  2. host = {id}
  • Start the daemon the manual way.:
  1. ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f]
  • Start the daemon the right way (using ceph.conf entry).:
  1. service ceph start
  • If starting the daemon fails with this error:
  1. mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument

Then make sure you do not have a keyring set in ceph.conf in the global section; move it to the client section; or add a keyring setting specific to this mds daemon. And verify that you see the same key in the mds data directory and ceph auth get mds.{id} output.

Summary

Once you have your monitor and two OSDs up and running, you can watch theplacement groups peer by executing the following:

  1. ceph -w

To view the tree, execute the following:

  1. ceph osd tree

You should see output that looks something like this:

  1. # id    weight  type name       up/down reweight
  2. -1      2       root default
  3. -2      2               host node1
  4. 0       1                       osd.0   up      1
  5. -3      1               host node2
  6. 1       1                       osd.1   up      1

To add (or remove) additional monitors, see Add/Remove Monitors.To add (or remove) additional Ceph OSD Daemons, see Add/Remove OSDs.