Configuration Management System

The configuration management system exists to provide every daemon with theproper configuration information. The configuration can be viewed as a set ofkey-value pairs.

  • How can the configuration be set? Well, there are several sources:

    • the ceph configuration file, usually named ceph.conf

      • command line arguments::
      • –debug-ms=1–debug-pg=10etc.

    • arguments injected at runtime using “injectargs” or “config set”

The Configuration File

Most configuration settings originate in the Ceph configuration file.

  • How do we find the configuration file? Well, in order, we check:

    • the default locations

    • the environment variable CEPH_CONF

    • the command line argument -c

Each stanza of the configuration file describes the key-value pairs that will be ineffect for a particular subset of the daemons. The “global” stanza applies toeverything. The “mon”, “osd”, and “mds” stanzas specify settings to take effectfor all monitors, all OSDs, and all mds servers, respectively. A stanza of theform mon.$name, osd.$name, or mds.$name gives settings for the monitor, OSD, orMDS of that name, respectively. Configuration values that appear later in thefile win over earlier ones.

A sample configuration file can be found in src/sample.ceph.conf.

Metavariables

The configuration system allows any configuration value to besubstituted into another value using the $varname syntax, similarto how bash shell expansion works.

  • A few additional special metavariables are also defined:

    • $host: expands to the current hostname

    • $type: expands to one of “mds”, “osd”, “mon”, or “client”

    • $id: expands to the daemon identifier. For osd.0, this would be 0; for mds.a, it would be a; for client.admin, it would be admin.

    • $num: same as $id

    • $name: expands to $type.$id

Reading configuration values

There are two ways for Ceph code to get configuration values. One way is toread it directly from a variable named “g_conf,” or equivalently,“g_ceph_ctx->_conf.” The other is to register an observer that will be calledevery time the relevant configuration values changes. This observer will becalled soon after the initial configuration is read, and every time after thatwhen one of the relevant values changes. Each observer tracks a set of keysand is invoked only when one of the relevant keys changes.

The interface to implement is found in common/config_obs.h.

  • The observer method should be preferred in new code because

    • It is more flexible, allowing the code to do whatever reinitialization needsto be done to implement the new configuration value.

    • It is the only way to create a std::string configuration variable that canbe changed by injectargs.

    • Even for int-valued configuration options, changing the values in one threadwhile another thread is reading them can lead to subtle andimpossible-to-diagnose bugs.

For these reasons, reading directly from g_conf should be considered deprecatedand not done in new code. Do not ever alter g_conf.

Changing configuration values

Configuration values can be changed by calling g_conf()->set_val. After changingthe configuration, you should call g_conf()->apply_changes to re-run all theaffected configuration observers. For convenience, you can callg_conf()->set_val_or_die to make a configuration change which you think shouldnever fail.

Injectargs, parse_argv, and parse_env are three other functions which modifythe configuration. Just like with set_val, you should call apply_changes aftercalling these functions to make sure your changes get applied.

Defining config options

New-style config options are defined in common/options.cc. All new configoptions should go here (and not into legacy_config_opts.h).

Levels

The Option constructor takes a “level” value:

  • LEVEL_BASIC is for basic config options that a normal operator is likely to adjust.

  • LEVEL_ADVANCED is for options that an operator can adjust, but should not touch unless they understand what they are doing. Adjusting advanced options poorly can lead to problems (performance or even data loss) if done incorrectly.

  • LEVEL_DEV is for options in place for use by developers only, either for testing purposes, or to describe constants that no user should adjust but we prefer not to compile into the code.

Description and long description

Short description of the option. Sentence fragment. e.g.:

  1. .set_description("Default checksum algorithm to use")

The long description is complete sentences, perhaps even multipleparagraphs, and may include other detailed information or notes.:

  1. .set_long_description("crc32c, xxhash32, and xxhash64 are available.  The _16 and _8 variants use only a subset of the bits for more compact (but less reliable) checksumming.")

Default values

There is a default value for every config option. In some cases, there mayalso be a daemon default that only applies to code that declares itselfas a daemon (in this case, the regular default only applies to non-daemons).

Safety

If an option can be safely changed at runtime:

  1. .set_safe()

Service

Service is a component name, like “common”, “osd”, “rgw”, “mds”, etc. It maybe a list of components, like:

  1. .add_service("mon mds osd mgr")

For example, the rocksdb options affect both the osd and mon.

Tags

Tags identify options across services that relate in some way. Example include;

  • network – options affecting network configuration

  • mkfs – options that only matter at mkfs time

Enums

For options with a defined set of allowed values:

  1. .set_enum_allowed({"none", "crc32c", "crc32c_16", "crc32c_8", "xxhash32", "xxhash64"})

Flags

  • RUNTIME: the value can be updated at runtime

  • NO_MON_UPDATE: Daemons/clients do not pull this value from the monitor config database. We disallow setting this option via ‘ceph config set …’. This option should be configured via ceph.conf or via the command line.

  • STARTUP: option takes effect only during daemon startup

  • CLUSTER_CREATE: option only affects cluster creation

  • CREATE: option only affects daemon creation