Preflight Checklist

The ceph-deploy tool operates out of a directory on an adminnode. Any host with network connectivity and a modern pythonenvironment and ssh (such as Linux) should work.

In the descriptions below, Node refers to a single machine.

Preflight - 图1

Ceph-deploy Setup

Add Ceph repositories to the ceph-deploy admin node. Then, installceph-deploy.

Debian/Ubuntu

For Debian and Ubuntu distributions, perform the following steps:

  • Add the release key:
  1. wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
  • Add the Ceph packages to your repository. Use the command below andreplace {ceph-stable-release} with a stable Ceph release (e.g.,luminous.) For example:
  1. echo deb https://download.ceph.com/debian-{ceph-stable-release}/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
  • Update your repository and install ceph-deploy:
  1. sudo apt update
  2. sudo apt install ceph-deploy

Note

You can also use the EU mirror eu.ceph.com for downloading your packages by replacing https://ceph.com/ by http://eu.ceph.com/

RHEL/CentOS

For CentOS 7, perform the following steps:

  • On Red Hat Enterprise Linux 7, register the target machine withsubscription-manager, verify your subscriptions, and enable the“Extras” repository for package dependencies. For example:
  1. sudo subscription-manager repos --enable=rhel-7-server-extras-rpms
  • Install and enable the Extra Packages for Enterprise Linux (EPEL)repository:
  1. sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Please see the EPEL wiki page for more information.

  • Add the Ceph repository to your yum configuration file at /etc/yum.repos.d/ceph.repo with the following command. Replace {ceph-stable-release} with a stable Ceph release (e.g.,luminous.) For example:
  1. cat << EOM > /etc/yum.repos.d/ceph.repo
  2. [ceph-noarch]
  3. name=Ceph noarch packages
  4. baseurl=https://download.ceph.com/rpm-{ceph-stable-release}/el7/noarch
  5. enabled=1
  6. gpgcheck=1
  7. type=rpm-md
  8. gpgkey=https://download.ceph.com/keys/release.asc
  9. EOM
  • Update your repository and install ceph-deploy:
  1. sudo yum update
  2. sudo yum install ceph-deploy

Note

You can also use the EU mirror eu.ceph.com for downloading your packages by replacing https://ceph.com/ by http://eu.ceph.com/

openSUSE

The Ceph project does not currently publish release RPMs for openSUSE, buta stable version of Ceph is included in the default update repository, soinstalling it is just a matter of:

  1. sudo zypper install ceph
  2. sudo zypper install ceph-deploy

If the distro version is out-of-date, open a bug athttps://bugzilla.opensuse.org/index.cgi and possibly try your luck with one ofthe following repositories:

  • Hammer:
  1. https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ahammer&package=ceph
  • Jewel:
  1. https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ajewel&package=ceph

Ceph Node Setup

The admin node must have password-less SSH access to Ceph nodes.When ceph-deploy logs in to a Ceph node as a user, that particularuser must have passwordless sudo privileges.

Install NTP

We recommend installing NTP on Ceph nodes (especially on Ceph Monitor nodes) toprevent issues arising from clock drift. See Clock for details.

On CentOS / RHEL, execute:

  1. sudo yum install ntp ntpdate ntp-doc

On Debian / Ubuntu, execute:

  1. sudo apt install ntpsec

or:

  1. sudo apt install chrony

Ensure that you enable the NTP service. Ensure that each Ceph Node uses thesame NTP time server. See NTP for details.

Install SSH Server

For ALL Ceph Nodes perform the following steps:

  • Install an SSH server (if necessary) on each Ceph Node:
  1. sudo apt install openssh-server

or:

  1. sudo yum install openssh-server
  • Ensure the SSH server is running on ALL Ceph Nodes.

Create a Ceph Deploy User

The ceph-deploy utility must login to a Ceph node as a userthat has passwordless sudo privileges, because it needs to installsoftware and configuration files without prompting for passwords.

Recent versions of ceph-deploy support a —username option so you canspecify any user that has password-less sudo (including root, althoughthis is NOT recommended). To use ceph-deploy —username {username}, theuser you specify must have password-less SSH access to the Ceph node, asceph-deploy will not prompt you for a password.

We recommend creating a specific user for ceph-deploy on ALL Ceph nodesin the cluster. Please do NOT use “ceph” as the user name. A uniform username across the cluster may improve ease of use (not required), but you shouldavoid obvious user names, because hackers typically use them with brute forcehacks (e.g., root, admin, {productname}). The following procedure,substituting {username} for the user name you define, describes how tocreate a user with passwordless sudo.

Note

Starting with the Infernalis release, the “ceph” user name is reservedfor the Ceph daemons. If the “ceph” user already exists on the Ceph nodes,removing the user must be done before attempting an upgrade.

  • Create a new user on each Ceph Node.
  1. ssh user@ceph-server
  2. sudo useradd -d /home/{username} -m {username}
  3. sudo passwd {username}
  • For the new user you added to each Ceph node, ensure that the user hassudo privileges.
  1. echo "{username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/{username}
  2. sudo chmod 0440 /etc/sudoers.d/{username}

Enable Password-less SSH

Since ceph-deploy will not prompt for a password, you must generateSSH keys on the admin node and distribute the public key to each Cephnode. ceph-deploy will attempt to generate the SSH keys for initialmonitors.

  • Generate the SSH keys, but do not use sudo or theroot user. Leave the passphrase empty:
  1. ssh-keygen
  2.  
  3. Generating public/private key pair.
  4. Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
  5. Enter passphrase (empty for no passphrase):
  6. Enter same passphrase again:
  7. Your identification has been saved in /ceph-admin/.ssh/id_rsa.
  8. Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
  1. ssh-copy-id {username}@node1
  2. ssh-copy-id {username}@node2
  3. ssh-copy-id {username}@node3
  • (Recommended) Modify the ~/.ssh/config file of your ceph-deployadmin node so that ceph-deploy can log in to Ceph nodes as the user youcreated without requiring you to specify —username {username} eachtime you execute ceph-deploy. This has the added benefit of streamliningssh and scp usage. Replace {username} with the user name youcreated:
  1. Host node1
  2.    Hostname node1
  3.    User {username}
  4. Host node2
  5.    Hostname node2
  6.    User {username}
  7. Host node3
  8.    Hostname node3
  9.    User {username}

Enable Networking On Bootup

Ceph OSDs peer with each other and report to Ceph Monitors over the network.If networking is off by default, the Ceph cluster cannot come onlineduring bootup until you enable networking.

The default configuration on some distributions (e.g., CentOS) has thenetworking interface(s) off by default. Ensure that, during boot up, yournetwork interface(s) turn(s) on so that your Ceph daemons can communicate overthe network. For example, on Red Hat and CentOS, navigate to/etc/sysconfig/network-scripts and ensure that the ifcfg-{iface} filehas ONBOOT set to yes.

Ensure Connectivity

Ensure connectivity using ping with short hostnames (hostname -s).Address hostname resolution issues as necessary.

Note

Hostnames should resolve to a network IP address, not to theloopback IP address (e.g., hostnames should resolve to an IP address otherthan 127.0.0.1). If you use your admin node as a Ceph node, youshould also ensure that it resolves to its hostname and IP address(i.e., not its loopback IP address).

Open Required Ports

Ceph Monitors communicate using port 6789 by default. Ceph OSDs communicatein a port range of 6800:7300 by default. See the Network ConfigurationReference for details. Ceph OSDs can use multiple network connections tocommunicate with clients, monitors, other OSDs for replication, and other OSDsfor heartbeats.

On some distributions (e.g., RHEL), the default firewall configuration is fairlystrict. You may need to adjust your firewall settings allow inbound requests sothat clients in your network can communicate with daemons on your Ceph nodes.

For firewalld on RHEL 7, add the ceph-mon service for Ceph Monitornodes and the ceph service for Ceph OSDs and MDSs to the public zone andensure that you make the settings permanent so that they are enabled on reboot.

For example, on monitors:

  1. sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent

and on OSDs and MDSs:

  1. sudo firewall-cmd --zone=public --add-service=ceph --permanent

Once you have finished configuring firewalld with the —permanent flag, you can make the changes live immediately without rebooting:

  1. sudo firewall-cmd --reload

For iptables, add port 6789 for Ceph Monitors and ports 6800:7300for Ceph OSDs. For example:

  1. sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT

Once you have finished configuring iptables, ensure that you make thechanges persistent on each node so that they will be in effect when your nodesreboot. For example:

  1. /sbin/service iptables save

TTY

On CentOS and RHEL, you may receive an error while trying to executeceph-deploy commands. If requiretty is set by default on your Cephnodes, disable it by executing sudo visudo and locate the Defaultsrequiretty setting. Change it to Defaults:ceph !requiretty or comment itout to ensure that ceph-deploy can connect using the user you created withCreate a Ceph Deploy User.

Note

If editing, /etc/sudoers, ensure that you usesudo visudo rather than a text editor.

SELinux

On CentOS and RHEL, SELinux is set to Enforcing by default. To streamline yourinstallation, we recommend setting SELinux to Permissive or disabling itentirely and ensuring that your installation and cluster are working properlybefore hardening your configuration. To set SELinux to Permissive, execute thefollowing:

  1. sudo setenforce 0

To configure SELinux persistently (recommended if SELinux is an issue), modifythe configuration file at /etc/selinux/config.

Priorities/Preferences

Ensure that your package manager has priority/preferences packages installed andenabled. On CentOS, you may need to install EPEL. On RHEL, you may need toenable optional repositories.

  1. sudo yum install yum-plugin-priorities

For example, on RHEL 7 server, execute the following to installyum-plugin-priorities and enable the rhel-7-server-optional-rpmsrepository:

  1. sudo yum install yum-plugin-priorities --enablerepo=rhel-7-server-optional-rpms

Summary

This completes the Quick Start Preflight. Proceed to the Storage ClusterQuick Start.