Monitor Config Reference

Understanding how to configure a Ceph Monitor is an important part ofbuilding a reliable Ceph Storage Cluster. All Ceph Storage Clustershave at least one monitor. A monitor configuration usually remains fairlyconsistent, but you can add, remove or replace a monitor in a cluster. SeeAdding/Removing a Monitor and Add/Remove a Monitor (ceph-deploy) fordetails.

Background

Ceph Monitors maintain a “master copy” of the cluster map, which means aCeph Client can determine the location of all Ceph Monitors, Ceph OSDDaemons, and Ceph Metadata Servers just by connecting to one Ceph Monitor andretrieving a current cluster map. Before Ceph Clients can read from or write toCeph OSD Daemons or Ceph Metadata Servers, they must connect to a Ceph Monitorfirst. With a current copy of the cluster map and the CRUSH algorithm, a CephClient can compute the location for any object. The ability to compute objectlocations allows a Ceph Client to talk directly to Ceph OSD Daemons, which is avery important aspect of Ceph’s high scalability and performance. SeeScalability and High Availability for additional details.

The primary role of the Ceph Monitor is to maintain a master copy of the clustermap. Ceph Monitors also provide authentication and logging services. CephMonitors write all changes in the monitor services to a single Paxos instance,and Paxos writes the changes to a key/value store for strong consistency. CephMonitors can query the most recent version of the cluster map during syncoperations. Ceph Monitors leverage the key/value store’s snapshots and iterators(using leveldb) to perform store-wide synchronization.

Deprecated since version version: 0.58

In Ceph versions 0.58 and earlier, Ceph Monitors use a Paxos instance foreach service and store the map as a file.

Cluster Maps

The cluster map is a composite of maps, including the monitor map, the OSD map,the placement group map and the metadata server map. The cluster map tracks anumber of important things: which processes are in the Ceph Storage Cluster;which processes that are in the Ceph Storage Cluster are up and runningor down; whether, the placement groups are active or inactive, andclean or in some other state; and, other details that reflect the currentstate of the cluster such as the total amount of storage space, and the amountof storage used.

When there is a significant change in the state of the cluster–e.g., a Ceph OSDDaemon goes down, a placement group falls into a degraded state, etc.–thecluster map gets updated to reflect the current state of the cluster.Additionally, the Ceph Monitor also maintains a history of the prior states ofthe cluster. The monitor map, OSD map, placement group map and metadata servermap each maintain a history of their map versions. We call each version an“epoch.”

When operating your Ceph Storage Cluster, keeping track of these states is animportant part of your system administration duties. See Monitoring a Clusterand Monitoring OSDs and PGs for additional details.

Monitor Quorum

Our Configuring ceph section provides a trivial Ceph configuration file thatprovides for one monitor in the test cluster. A cluster will run fine with asingle monitor; however, a single monitor is a single-point-of-failure. Toensure high availability in a production Ceph Storage Cluster, you should runCeph with multiple monitors so that the failure of a single monitor WILL NOTbring down your entire cluster.

When a Ceph Storage Cluster runs multiple Ceph Monitors for high availability,Ceph Monitors use Paxos) to establish consensus about the master cluster map.A consensus requires a majority of monitors running to establish a quorum forconsensus about the cluster map (e.g., 1; 2 out of 3; 3 out of 5; 4 out of 6;etc.).

mon force quorum join

• Description

• Force monitor to join quorum even if it has been previously removed from the map

• Type

• Boolean

• Default

• False

Consistency

When you add monitor settings to your Ceph configuration file, you need to beaware of some of the architectural aspects of Ceph Monitors. Ceph imposesstrict consistency requirements for a Ceph monitor when discovering anotherCeph Monitor within the cluster. Whereas, Ceph Clients and other Ceph daemonsuse the Ceph configuration file to discover monitors, monitors discover eachother using the monitor map (monmap), not the Ceph configuration file.

A Ceph Monitor always refers to the local copy of the monmap when discoveringother Ceph Monitors in the Ceph Storage Cluster. Using the monmap instead of theCeph configuration file avoids errors that could break the cluster (e.g., typosin ceph.conf when specifying a monitor address or port). Since monitors usemonmaps for discovery and they share monmaps with clients and other Cephdaemons, the monmap provides monitors with a strict guarantee that theirconsensus is valid.

Strict consistency also applies to updates to the monmap. As with any otherupdates on the Ceph Monitor, changes to the monmap always run through adistributed consensus algorithm called Paxos). The Ceph Monitors must agree oneach update to the monmap, such as adding or removing a Ceph Monitor, to ensurethat each monitor in the quorum has the same version of the monmap. Updates tothe monmap are incremental so that Ceph Monitors have the latest agreed uponversion, and a set of previous versions. Maintaining a history enables a CephMonitor that has an older version of the monmap to catch up with the currentstate of the Ceph Storage Cluster.

If Ceph Monitors discovered each other through the Ceph configuration fileinstead of through the monmap, it would introduce additional risks because theCeph configuration files are not updated and distributed automatically. CephMonitors might inadvertently use an older Ceph configuration file, fail torecognize a Ceph Monitor, fall out of a quorum, or develop a situation wherePaxos) is not able to determine the current state of the system accurately.

Bootstrapping Monitors

In most configuration and deployment cases, tools that deploy Ceph may helpbootstrap the Ceph Monitors by generating a monitor map for you (e.g.,ceph-deploy, etc). A Ceph Monitor requires a few explicitsettings:

• Filesystem ID: The fsid is the unique identifier for yourobject store. Since you can run multiple clusters on the samehardware, you must specify the unique ID of the object store whenbootstrapping a monitor. Deployment tools usually do this for you(e.g., ceph-deploy can call a tool like uuidgen), but youmay specify the fsid manually too.

• Monitor ID: A monitor ID is a unique ID assigned to each monitor withinthe cluster. It is an alphanumeric value, and by convention the identifierusually follows an alphabetical increment (e.g., a, b, etc.). Thiscan be set in a Ceph configuration file (e.g., [mon.a], [mon.b], etc.),by a deployment tool, or using the ceph commandline.

• Keys: The monitor must have secret keys. A deployment tool such asceph-deploy usually does this for you, but you mayperform this step manually too. See Monitor Keyrings for details.

For additional details on bootstrapping, see Bootstrapping a Monitor.

Configuring Monitors

To apply configuration settings to the entire cluster, enter the configurationsettings under [global]. To apply configuration settings to all monitors inyour cluster, enter the configuration settings under [mon]. To applyconfiguration settings to specific monitors, specify the monitor instance(e.g., [mon.a]). By convention, monitor instance names use alpha notation.

[global]
 
[mon]
 
[mon.a]
 
[mon.b]
 
[mon.c]

Minimum Configuration

The bare minimum monitor settings for a Ceph monitor via the Ceph configurationfile include a hostname and a monitor address for each monitor. You can configurethese under [mon] or under the entry for a specific monitor.

[global]
        mon host = 10.0.0.2,10.0.0.3,10.0.0.4

[mon.a]
        host = hostname1
        mon addr = 10.0.0.10:6789

See the Network Configuration Reference for details.

Note

This minimum configuration for monitors assumes that a deploymenttool generates the fsid and the mon. key for you.

Once you deploy a Ceph cluster, you SHOULD NOT change the IP address ofthe monitors. However, if you decide to change the monitor’s IP address, youmust follow a specific procedure. See Changing a Monitor’s IP Address fordetails.

Monitors can also be found by clients using DNS SRV records. See Monitor lookup through DNS for details.

Cluster ID

Each Ceph Storage Cluster has a unique identifier (fsid). If specified, itusually appears under the [global] section of the configuration file.Deployment tools usually generate the fsid and store it in the monitor map,so the value may not appear in a configuration file. The fsid makes itpossible to run daemons for multiple clusters on the same hardware.

fsid

• Description

• The cluster ID. One per cluster.

• Type

• UUID

• Required

• Yes.

• Default

• N/A. May be generated by a deployment tool if not specified.

Note

Do not set this value if you use a deployment tool that doesit for you.

Initial Members

We recommend running a production Ceph Storage Cluster with at least three CephMonitors to ensure high availability. When you run multiple monitors, you mayspecify the initial monitors that must be members of the cluster in order toestablish a quorum. This may reduce the time it takes for your cluster to comeonline.

[mon]
        mon initial members = a,b,c

mon initial members

• Description

• The IDs of initial monitors in a cluster during startup. Ifspecified, Ceph requires an odd number of monitors to form aninitial quorum (e.g., 3).

• Type

• String

• Default

• None

Note

A majority of monitors in your cluster must be able to reacheach other in order to establish a quorum. You can decrease the initialnumber of monitors to establish a quorum with this setting.

Data

Ceph provides a default path where Ceph Monitors store data. For optimalperformance in a production Ceph Storage Cluster, we recommend running CephMonitors on separate hosts and drives from Ceph OSD Daemons. As leveldb is usingmmap() for writing the data, Ceph Monitors flush their data from memory to diskvery often, which can interfere with Ceph OSD Daemon workloads if the datastore is co-located with the OSD Daemons.

In Ceph versions 0.58 and earlier, Ceph Monitors store their data in files. Thisapproach allows users to inspect monitor data with common tools like lsand cat. However, it doesn’t provide strong consistency.

In Ceph versions 0.59 and later, Ceph Monitors store their data as key/valuepairs. Ceph Monitors require ACID transactions. Using a data store preventsrecovering Ceph Monitors from running corrupted versions through Paxos, and itenables multiple modification operations in one single atomic batch, among otheradvantages.

Generally, we do not recommend changing the default data location. If you modifythe default location, we recommend that you make it uniform across Ceph Monitorsby setting it in the [mon] section of the configuration file.

mon data

• Description

• The monitor’s data location.

• Type

• String

• Default

• /var/lib/ceph/mon/$cluster-$id

mon data size warn

• Description

• Issue a HEALTH_WARN in cluster log when the monitor’s datastore goes over 15GB.

• Type

• Integer

• Default

• 1510241024*1024

mon data avail warn

• Description

• Issue a HEALTH_WARN in cluster log when the available diskspace of monitor’s data store is lower or equal to thispercentage.

• Type

• Integer

• Default

• 30

mon data avail crit

• Description

• Issue a HEALTH_ERR in cluster log when the available diskspace of monitor’s data store is lower or equal to thispercentage.

• Type

• Integer

• Default

• 5

mon warn on cache pools without hit sets

• Description

• Issue a HEALTH_WARN in cluster log if a cache pool does nothave the hit_set_type value configured.See hit_set_type for moredetails.

• Type

• Boolean

• Default

• True

mon warn on crush straw calc version zero

• Description

• Issue a HEALTH_WARN in cluster log if the CRUSH’sstraw_calc_version is zero. SeeCRUSH map tunables fordetails.

• Type

• Boolean

• Default

• True

mon warn on legacy crush tunables

• Description

• Issue a HEALTH_WARN in cluster log ifCRUSH tunables are too old (older than mon_min_crush_required_version)

• Type

• Boolean

• Default

• True

mon crush min required version

• Description

• The minimum tunable profile version required by the cluster.SeeCRUSH map tunables fordetails.

• Type

• String

• Default

• hammer

mon warn on osd down out interval zero

• Description

• Issue a HEALTH_WARN in cluster log ifmon osd down out interval is zero. Having this option set tozero on the leader acts much like the noout flag. It’s hardto figure out what’s going wrong with clusters without thenoout flag set but acting like that just the same, so wereport a warning in this case.

• Type

• Boolean

• Default

• True

mon warn on slow ping ratio

• Description

• Issue a HEALTH_WARN in cluster log if any heartbeatbetween OSDs exceeds mon warn on slow ping ratioof osd heartbeat grace. The default is 5%.

• Type

• Float

• Default

• 0.05

mon warn on slow ping time

• Description

• Override mon warn on slow ping ratio with a specific value.Issue a HEALTH_WARN in cluster log if any heartbeatbetween OSDs exceeds mon warn on slow ping timemilliseconds. The default is 0 (disabled).

• Type

• Integer

• Default

• 0

mon warn on pool no redundancy

• Description

• Issue a HEALTH_WARN in cluster log if any pool isconfigured with no replicas.

• Type

• Boolean

• Default

• True

mon cache target full warn ratio

• Description

• Position between pool’s cache_target_full andtarget_max_object where we start warning

• Type

• Float

• Default

• 0.66

mon health to clog

• Description

• Enable sending health summary to cluster log periodically.

• Type

• Boolean

• Default

• True

mon health to clog tick interval

• Description

• How often (in seconds) the monitor send health summary to clusterlog (a non-positive number disables it). If current health summaryis empty or identical to the last time, monitor will not send itto cluster log.

• Type

• Float

• Default

• 60.0

mon health to clog interval

• Description

• How often (in seconds) the monitor send health summary to clusterlog (a non-positive number disables it). Monitor will alwayssend the summary to cluster log no matter if the summary changesor not.

• Type

• Integer

• Default

• 3600

Storage Capacity

When a Ceph Storage Cluster gets close to its maximum capacity (i.e., mon osdfull ratio), Ceph prevents you from writing to or reading from Ceph OSDDaemons as a safety measure to prevent data loss. Therefore, letting aproduction Ceph Storage Cluster approach its full ratio is not a good practice,because it sacrifices high availability. The default full ratio is .95, or95% of capacity. This a very aggressive setting for a test cluster with a smallnumber of OSDs.

Tip

When monitoring your cluster, be alert to warnings related to thenearfull ratio. This means that a failure of some OSDs could resultin a temporary service disruption if one or more OSDs fails. Consider addingmore OSDs to increase storage capacity.

A common scenario for test clusters involves a system administrator removing aCeph OSD Daemon from the Ceph Storage Cluster to watch the cluster rebalance;then, removing another Ceph OSD Daemon, and so on until the Ceph Storage Clustereventually reaches the full ratio and locks up. We recommend a bit of capacityplanning even with a test cluster. Planning enables you to gauge how much sparecapacity you will need in order to maintain high availability. Ideally, you wantto plan for a series of Ceph OSD Daemon failures where the cluster can recoverto an active + clean state without replacing those Ceph OSD Daemonsimmediately. You can run a cluster in an active + degraded state, but thisis not ideal for normal operating conditions.

The following diagram depicts a simplistic Ceph Storage Cluster containing 33Ceph Nodes with one Ceph OSD Daemon per host, each Ceph OSD Daemon reading fromand writing to a 3TB drive. So this exemplary Ceph Storage Cluster has a maximumactual capacity of 99TB. With a mon osd full ratio of 0.95, if the CephStorage Cluster falls to 5TB of remaining capacity, the cluster will not allowCeph Clients to read and write data. So the Ceph Storage Cluster’s operatingcapacity is 95TB, not 99TB.

It is normal in such a cluster for one or two OSDs to fail. A less frequent butreasonable scenario involves a rack’s router or power supply failing, whichbrings down multiple OSDs simultaneously (e.g., OSDs 7-12). In such a scenario,you should still strive for a cluster that can remain operational and achieve anactive + clean state–even if that means adding a few hosts with additionalOSDs in short order. If your capacity utilization is too high, you may not losedata, but you could still sacrifice data availability while resolving an outagewithin a failure domain if capacity utilization of the cluster exceeds the fullratio. For this reason, we recommend at least some rough capacity planning.

Identify two numbers for your cluster:

• The number of OSDs.

• The total capacity of the cluster

If you divide the total capacity of your cluster by the number of OSDs in yourcluster, you will find the mean average capacity of an OSD within your cluster.Consider multiplying that number by the number of OSDs you expect will failsimultaneously during normal operations (a relatively small number). Finallymultiply the capacity of the cluster by the full ratio to arrive at a maximumoperating capacity; then, subtract the number of amount of data from the OSDsyou expect to fail to arrive at a reasonable full ratio. Repeat the foregoingprocess with a higher number of OSD failures (e.g., a rack of OSDs) to arrive ata reasonable number for a near full ratio.

The following settings only apply on cluster creation and are then stored inthe OSDMap.

[global]
 
        mon osd full ratio = .80
        mon osd backfillfull ratio = .75
        mon osd nearfull ratio = .70

mon osd full ratio

• Description

• The percentage of disk space used before an OSD isconsidered full.

• Type

• Float

• Default

• 0.95

mon osd backfillfull ratio

• Description

• The percentage of disk space used before an OSD isconsidered too full to backfill.

• Type

• Float

• Default

• 0.90

mon osd nearfull ratio

• Description

• The percentage of disk space used before an OSD isconsidered nearfull.

• Type

• Float

• Default

• 0.85

Tip

If some OSDs are nearfull, but others have plenty of capacity, youmay have a problem with the CRUSH weight for the nearfull OSDs.

Tip

These settings only apply during cluster creation. Afterwards they needto be changed in the OSDMap using ceph osd set-nearfull-ratio andceph osd set-full-ratio

Heartbeat

Ceph monitors know about the cluster by requiring reports from each OSD, and byreceiving reports from OSDs about the status of their neighboring OSDs. Cephprovides reasonable default settings for monitor/OSD interaction; however, youmay modify them as needed. See Monitor/OSD Interaction for details.

Monitor Store Synchronization

When you run a production cluster with multiple monitors (recommended), eachmonitor checks to see if a neighboring monitor has a more recent version of thecluster map (e.g., a map in a neighboring monitor with one or more epoch numbershigher than the most current epoch in the map of the instant monitor).Periodically, one monitor in the cluster may fall behind the other monitors tothe point where it must leave the quorum, synchronize to retrieve the mostcurrent information about the cluster, and then rejoin the quorum. For thepurposes of synchronization, monitors may assume one of three roles:

• Leader: The Leader is the first monitor to achieve the most recentPaxos version of the cluster map.

• Provider: The Provider is a monitor that has the most recent versionof the cluster map, but wasn’t the first to achieve the most recent version.

• Requester: A Requester is a monitor that has fallen behind the leaderand must synchronize in order to retrieve the most recent information aboutthe cluster before it can rejoin the quorum.

These roles enable a leader to delegate synchronization duties to a provider,which prevents synchronization requests from overloading the leader–improvingperformance. In the following diagram, the requester has learned that it hasfallen behind the other monitors. The requester asks the leader to synchronize,and the leader tells the requester to synchronize with a provider.

Synchronization always occurs when a new monitor joins the cluster. Duringruntime operations, monitors may receive updates to the cluster map at differenttimes. This means the leader and provider roles may migrate from one monitor toanother. If this happens while synchronizing (e.g., a provider falls behind theleader), the provider can terminate synchronization with a requester.

Once synchronization is complete, Ceph requires trimming across the cluster.Trimming requires that the placement groups are active + clean.

mon sync timeout

• Description

• Number of seconds the monitor will wait for the next updatemessage from its sync provider before it gives up and bootstrapagain.

• Type

• Double

• Default

• 60.0

mon sync max payload size

• Description

• The maximum size for a sync payload (in bytes).

• Type

• 32-bit Integer

• Default

• 1048576

paxos max join drift

• Description

• The maximum Paxos iterations before we must first sync themonitor data stores. When a monitor finds that its peer is toofar ahead of it, it will first sync with data stores before movingon.

• Type

• Integer

• Default

• 10

paxos stash full interval

• Description

• How often (in commits) to stash a full copy of the PaxosService state.Current this setting only affects mds, mon, auth and mgrPaxosServices.

• Type

• Integer

• Default

• 25

paxos propose interval

• Description

• Gather updates for this time interval before proposinga map update.

• Type

• Double

• Default

• 1.0

paxos min

• Description

• The minimum number of paxos states to keep around

• Type

• Integer

• Default

• 500

paxos min wait

• Description

• The minimum amount of time to gather updates after a period ofinactivity.

• Type

• Double

• Default

• 0.05

paxos trim min

• Description

• Number of extra proposals tolerated before trimming

• Type

• Integer

• Default

• 250

paxos trim max

• Description

• The maximum number of extra proposals to trim at a time

• Type

• Integer

• Default

• 500

paxos service trim min

• Description

• The minimum amount of versions to trigger a trim (0 disables it)

• Type

• Integer

• Default

• 250

paxos service trim max

• Description

• The maximum amount of versions to trim during a single proposal (0 disables it)

• Type

• Integer

• Default

• 500

mon mds force trim to

• Description

• Force monitor to trim mdsmaps to this point (0 disables it.dangerous, use with care)

• Type

• Integer

• Default

• 0

mon osd force trim to

• Description

• Force monitor to trim osdmaps to this point, even if there isPGs not clean at the specified epoch (0 disables it. dangerous,use with care)

• Type

• Integer

• Default

• 0

mon osd cache size

• Description

• The size of osdmaps cache, not to rely on underlying store’s cache

• Type

• Integer

• Default

• 500

mon election timeout

• Description

• On election proposer, maximum waiting time for all ACKs in seconds.

• Type

• Float

• Default

• 5.00

mon lease

• Description

• The length (in seconds) of the lease on the monitor’s versions.

• Type

• Float

• Default

• 5.00

mon lease renew interval factor

• Description

• mon lease * mon lease renew interval factor will be theinterval for the Leader to renew the other monitor’s leases. Thefactor should be less than 1.0.

• Type

• Float

• Default

• 0.60

mon lease ack timeout factor

• Description

• The Leader will wait mon lease * mon lease ack timeout factorfor the Providers to acknowledge the lease extension.

• Type

• Float

• Default

• 2.00

mon accept timeout factor

• Description

• The Leader will wait mon lease * mon accept timeout factorfor the Requester(s) to accept a Paxos update. It is also usedduring the Paxos recovery phase for similar purposes.

• Type

• Float

• Default

• 2.00

mon min osdmap epochs

• Description

• Minimum number of OSD map epochs to keep at all times.

• Type

• 32-bit Integer

• Default

• 500

mon max log epochs

• Description

• Maximum number of Log epochs the monitor should keep.

• Type

• 32-bit Integer

• Default

• 500

Clock

Ceph daemons pass critical messages to each other, which must be processedbefore daemons reach a timeout threshold. If the clocks in Ceph monitorsare not synchronized, it can lead to a number of anomalies. For example:

• Daemons ignoring received messages (e.g., timestamps outdated)

• Timeouts triggered too soon/late when a message wasn’t received in time.

See Monitor Store Synchronization for details.

Tip

You SHOULD install NTP on your Ceph monitor hosts toensure that the monitor cluster operates with synchronized clocks.

Clock drift may still be noticeable with NTP even though the discrepancy is notyet harmful. Ceph’s clock drift / clock skew warnings may get triggered eventhough NTP maintains a reasonable level of synchronization. Increasing yourclock drift may be tolerable under such circumstances; however, a number offactors such as workload, network latency, configuring overrides to defaulttimeouts and the Monitor Store Synchronization settings may influencethe level of acceptable clock drift without compromising Paxos guarantees.

Ceph provides the following tunable options to allow you to findacceptable values.

mon tick interval

• Description

• A monitor’s tick interval in seconds.

• Type

• 32-bit Integer

• Default

• 5

mon clock drift allowed

• Description

• The clock drift in seconds allowed between monitors.

• Type

• Float

• Default

• 0.05

mon clock drift warn backoff

• Description

• Exponential backoff for clock drift warnings

• Type

• Float

• Default

• 5.00

mon timecheck interval

• Description

• The time check interval (clock drift check) in secondsfor the Leader.

• Type

• Float

• Default

• 300.00

mon timecheck skew interval

• Description

• The time check interval (clock drift check) in seconds when inpresence of a skew in seconds for the Leader.

• Type

• Float

• Default

• 30.00

Client

mon client hunt interval

• Description

• The client will try a new monitor every N seconds until itestablishes a connection.

• Type

• Double

• Default

• 3.00

mon client ping interval

• Description

• The client will ping the monitor every N seconds.

• Type

• Double

• Default

• 10.00

mon client max log entries per message

• Description

• The maximum number of log entries a monitor will generateper client message.

• Type

• Integer

• Default

• 1000

mon client bytes

• Description

• The amount of client message data allowed in memory (in bytes).

• Type

• 64-bit Integer Unsigned

• Default

• 100ul << 20

Pool settings

Since version v0.94 there is support for pool flags which allow or disallow changes to be made to pools.

Monitors can also disallow removal of pools if configured that way.

mon allow pool delete

• Description

• If the monitors should allow pools to be removed. Regardless of what the pool flags say.

• Type

• Boolean

• Default

• false

osd pool default ec fast read

• Description

• Whether to turn on fast read on the pool or not. It will be used asthe default setting of newly created erasure coded pools if fast_readis not specified at create time.

• Type

• Boolean

• Default

• false

osd pool default flag hashpspool

• Description

• Set the hashpspool flag on new pools

• Type

• Boolean

• Default

• true

osd pool default flag nodelete

• Description

• Set the nodelete flag on new pools. Prevents allow pool removal with this flag in any way.

• Type

• Boolean

• Default

• false

osd pool default flag nopgchange

• Description

• Set the nopgchange flag on new pools. Does not allow the number of PGs to be changed for a pool.

• Type

• Boolean

• Default

• false

osd pool default flag nosizechange

• Description

• Set the nosizechange flag on new pools. Does not allow the size to be changed of pool.

• Type

• Boolean

• Default

• false

For more information about the pool flags see Pool values.

Miscellaneous

mon max osd

• Description

• The maximum number of OSDs allowed in the cluster.

• Type

• 32-bit Integer

• Default

• 10000

mon globalid prealloc

• Description

• The number of global IDs to pre-allocate for clients and daemons in the cluster.

• Type

• 32-bit Integer

• Default

• 10000

mon subscribe interval

• Description

• The refresh interval (in seconds) for subscriptions. Thesubscription mechanism enables obtaining the cluster mapsand log information.

• Type

• Double

• Default

• 86400.00

mon stat smooth intervals

• Description

• Ceph will smooth statistics over the last N PG maps.

• Type

• Integer

• Default

• 6

mon probe timeout

• Description

• Number of seconds the monitor will wait to find peers before bootstrapping.

• Type

• Double

• Default

• 2.00

mon daemon bytes

• Description

• The message memory cap for metadata server and OSD messages (in bytes).

• Type

• 64-bit Integer Unsigned

• Default

• 400ul << 20

mon max log entries per event

• Description

• The maximum number of log entries per event.

• Type

• Integer

• Default

• 4096

mon osd prime pg temp

• Description

• Enables or disable priming the PGMap with the previous OSDs when an outOSD comes back into the cluster. With the true setting the clientswill continue to use the previous OSDs until the newly in OSDs as thatPG peered.

• Type

• Boolean

• Default

• true

mon osd prime pg temp max time

• Description

• How much time in seconds the monitor should spend trying to prime thePGMap when an out OSD comes back into the cluster.

• Type

• Float

• Default

• 0.50

mon osd prime pg temp max time estimate

• Description

• Maximum estimate of time spent on each PG before we prime all PGsin parallel.

• Type

• Float

• Default

• 0.25

mon mds skip sanity

• Description

• Skip safety assertions on FSMap (in case of bugs where we want tocontinue anyway). Monitor terminates if the FSMap sanity checkfails, but we can disable it by enabling this option.

• Type

• Boolean

• Default

• False

mon max mdsmap epochs

• Description

• The maximum amount of mdsmap epochs to trim during a single proposal.

• Type

• Integer

• Default

• 500

mon config key max entry size

• Description

• The maximum size of config-key entry (in bytes)

• Type

• Integer

• Default

• 65536

mon scrub interval

• Description

• How often (in seconds) the monitor scrub its store by comparingthe stored checksums with the computed ones of all the storedkeys.

• Type

• Integer

• Default

• 3600*24

mon scrub max keys

• Description

• The maximum number of keys to scrub each time.

• Type

• Integer

• Default

• 100

mon compact on start

• Description

• Compact the database used as Ceph Monitor store onceph-mon start. A manual compaction helps to shrink themonitor database and improve the performance of it if the regularcompaction fails to work.

• Type

• Boolean

• Default

• False

mon compact on bootstrap

• Description

• Compact the database used as Ceph Monitor store onon bootstrap. Monitor starts probing each other for creatinga quorum after bootstrap. If it times out before joining thequorum, it will start over and bootstrap itself again.

• Type

• Boolean

• Default

• False

mon compact on trim

• Description

• Compact a certain prefix (including paxos) when we trim its old states.

• Type

• Boolean

• Default

• True

mon cpu threads

• Description

• Number of threads for performing CPU intensive work on monitor.

• Type

• Integer

• Default

• 4

mon osd mapping pgs per chunk

• Description

• We calculate the mapping from placement group to OSDs in chunks.This option specifies the number of placement groups per chunk.

• Type

• Integer

• Default

• 4096

mon session timeout

• Description

• Monitor will terminate inactive sessions stay idle over thistime limit.

• Type

• Integer

• Default

• 300

mon osd cache size min

• Description

• The minimum amount of bytes to be kept mapped in memory for osdmonitor caches.

• Type

• 64-bit Integer

• Default

• 134217728

mon memory target

• Description

• The amount of bytes pertaining to osd monitor caches and kv cacheto be kept mapped in memory with cache auto-tuning enabled.

• Type

• 64-bit Integer

• Default

• 2147483648

mon memory autotune

• Description

• Autotune the cache memory being used for osd monitors and kvdatabase.

• Type

• Boolean

• Default

• True