Logging and Debugging

Typically, when you add debugging to your Ceph configuration, you do so atruntime. You can also add Ceph debug logging to your Ceph configuration file ifyou are encountering issues when starting your cluster. You may view Ceph logfiles under /var/log/ceph (the default location).

Tip

When debug output slows down your system, the latency can hiderace conditions.

Logging is resource intensive. If you are encountering a problem in a specificarea of your cluster, enable logging for that area of the cluster. For example,if your OSDs are running fine, but your metadata servers are not, you shouldstart by enabling debug logging for the specific metadata server instance(s)giving you trouble. Enable logging for each subsystem as needed.

Important

Verbose logging can generate over 1GB of data per hour. If yourOS disk reaches its capacity, the node will stop working.

If you enable or increase the rate of Ceph logging, ensure that you havesufficient disk space on your OS disk. See Accelerating Log Rotation fordetails on rotating log files. When your system is running well, removeunnecessary debugging settings to ensure your cluster runs optimally. Loggingdebug output messages is relatively slow, and a waste of resources whenoperating your cluster.

See Subsystem, Log and Debug Settings for details on available settings.

Runtime

If you would like to see the configuration settings at runtime, you must login to a host with a running daemon and execute the following:

ceph daemon {daemon-name} config show | less

For example,:

ceph daemon osd.0 config show | less

To activate Ceph’s debugging output (i.e., dout()) at runtime, use theceph tell command to inject arguments into the runtime configuration:

ceph tell {daemon-type}.{daemon id or *} config set {name} {value}

Replace {daemon-type} with one of osd, mon or mds. You may applythe runtime setting to all daemons of a particular type with *, or specifya specific daemon’s ID. For example, to increasedebug logging for a ceph-osd daemon named osd.0, execute the following:

ceph tell osd.0 config set debug_osd 0/5

The ceph tell command goes through the monitors. If you cannot bind to themonitor, you can still make the change by logging into the host of the daemonwhose configuration you’d like to change using ceph daemon.For example:

sudo ceph daemon osd.0 config set debug_osd 0/5

See Subsystem, Log and Debug Settings for details on available settings.

Boot Time

To activate Ceph’s debugging output (i.e., dout()) at boot time, you mustadd settings to your Ceph configuration file. Subsystems common to each daemonmay be set under [global] in your configuration file. Subsystems forparticular daemons are set under the daemon section in your configuration file(e.g., [mon], [osd], [mds]). For example:

[global]
        debug ms = 1/5
 
[mon]
        debug mon = 20
        debug paxos = 1/5
        debug auth = 2
 
[osd]
        debug osd = 1/5
        debug filestore = 1/5
        debug journal = 1
        debug monc = 5/20
 
[mds]
        debug mds = 1
        debug mds balancer = 1

See Subsystem, Log and Debug Settings for details.

Accelerating Log Rotation

If your OS disk is relatively full, you can accelerate log rotation by modifyingthe Ceph log rotation file at /etc/logrotate.d/ceph. Add a size settingafter the rotation frequency to accelerate log rotation (via cronjob) if yourlogs exceed the size setting. For example, the default setting looks likethis:

rotate 7
weekly
compress
sharedscripts

Modify it by adding a size setting.

rotate 7
weekly
size 500M
compress
sharedscripts

Then, start the crontab editor for your user space.

crontab -e

Finally, add an entry to check the etc/logrotate.d/ceph file.

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1

The preceding example checks the etc/logrotate.d/ceph file every 30 minutes.

Valgrind

Debugging may also require you to track down memory and threading issues.You can run a single daemon, a type of daemon, or the whole cluster withValgrind. You should only use Valgrind when developing or debugging Ceph.Valgrind is computationally expensive, and will slow down your system otherwise.Valgrind messages are logged to stderr.

Subsystem, Log and Debug Settings

In most cases, you will enable debug logging output via subsystems.

Ceph Subsystems

Each subsystem has a logging level for its output logs, and for its logsin-memory. You may set different values for each of these subsystems by settinga log file level and a memory level for debug logging. Ceph’s logging levelsoperate on a scale of 1 to 20, where 1 is terse and 20 isverbose 1 . In general, the logs in-memory are not sent to the output log unless:

• a fatal signal is raised or

• an assert in source code is triggered or

• upon requested. Please consult document on admin socket for more details.

A debug logging setting can take a single value for the log level and thememory level, which sets them both as the same value. For example, if youspecify debug ms = 5, Ceph will treat it as a log level and a memory levelof 5. You may also specify them separately. The first setting is the loglevel, and the second setting is the memory level. You must separate them witha forward slash (/). For example, if you want to set the ms subsystem’sdebug logging level to 1 and its memory level to 5, you would specify itas debug ms = 1/5. For example:

debug {subsystem} = {log-level}/{memory-level}
#for example
debug mds balancer = 1/20

The following table provides a list of Ceph subsystems and their default log andmemory levels. Once you complete your logging efforts, restore the subsystemsto their default level or to a level suitable for normal operations.

Logging Settings

Logging and debugging settings are not required in a Ceph configuration file,but you may override default settings as needed. Ceph supports the followingsettings:

log file

• Description

• The location of the logging file for your cluster.

• Type

• String

• Required

• No

• Default

• /var/log/ceph/$cluster-$name.log

log max new

• Description

• The maximum number of new log files.

• Type

• Integer

• Required

• No

• Default

• 1000

log max recent

• Description

• The maximum number of recent events to include in a log file.

• Type

• Integer

• Required

• No

• Default

• 10000

log to stderr

• Description

• Determines if logging messages should appear in stderr.

• Type

• Boolean

• Required

• No

• Default

• true

err to stderr

• Description

• Determines if error messages should appear in stderr.

• Type

• Boolean

• Required

• No

• Default

• true

log to syslog

• Description

• Determines if logging messages should appear in syslog.

• Type

• Boolean

• Required

• No

• Default

• false

err to syslog

• Description

• Determines if error messages should appear in syslog.

• Type

• Boolean

• Required

• No

• Default

• false

log flush on exit

• Description

• Determines if Ceph should flush the log files after exit.

• Type

• Boolean

• Required

• No

• Default

• true

clog to monitors

• Description

• Determines if clog messages should be sent to monitors.

• Type

• Boolean

• Required

• No

• Default

• true

clog to syslog

• Description

• Determines if clog messages should be sent to syslog.

• Type

• Boolean

• Required

• No

• Default

• false

mon cluster log to syslog

• Description

• Determines if the cluster log should be output to the syslog.

• Type

• Boolean

• Required

• No

• Default

• false

mon cluster log file

• Description

• The locations of the cluster’s log files. There are two channels inCeph: cluster and audit. This option represents a mappingfrom channels to log files, where the log entries of thatchannel are sent to. The default entry is a fallbackmapping for channels not explicitly specified. So, the followingdefault setting will send cluster log to $cluster.log, andsend audit log to $cluster.audit.log, where $cluster willbe replaced with the actual cluster name.

• Type

• String

• Required

• No

• Default

• default=/var/log/ceph/$cluster.$channel.log,cluster=/var/log/ceph/$cluster.log

OSD

osd debug drop ping probability

• Description

• ?

• Type

• Double

• Required

• No

• Default

• 0

osd debug drop ping duration

• Description

• Type

• Integer

• Required

• No

• Default

• 0

osd debug drop pg create probability

• Description

• Type

• Integer

• Required

• No

• Default

• 0

osd debug drop pg create duration

• Description

• ?

• Type

• Double

• Required

• No

• Default

• 1

osd min pg log entries

• Description

• The minimum number of log entries for placement groups.

• Type

• 32-bit Unsigned Integer

• Required

• No

• Default

• 1000

osd op log threshold

• Description

• How many op log messages to show up in one pass.

• Type

• Integer

• Required

• No

• Default

• 5

Filestore

filestore debug omap check

• Description

• Debugging check on synchronization. This is an expensive operation.

• Type

• Boolean

• Required

• No

• Default

• false

MDS

mds debug scatterstat

• Description

• Ceph will assert that various recursive stat invariants are true(for developers only).

• Type

• Boolean

• Required

• No

• Default

• false

mds debug frag

• Description

• Ceph will verify directory fragmentation invariants whenconvenient (developers only).

• Type

• Boolean

• Required

• No

• Default

• false

mds debug auth pins

• Description

• The debug auth pin invariants (for developers only).

• Type

• Boolean

• Required

• No

• Default

• false

mds debug subtrees

• Description

• The debug subtree invariants (for developers only).

• Type

• Boolean

• Required

• No

• Default

• false

RADOS Gateway

rgw log nonexistent bucket

• Description

• Should we log a non-existent buckets?

• Type

• Boolean

• Required

• No

• Default

• false

rgw log object name

• Description

• Should an object’s name be logged. // man date to see codes (a subset are supported)

• Type

• String

• Required

• No

• Default

• %Y-%m-%d-%H-%i-%n

rgw log object name utc

• Description

• Object log name contains UTC?

• Type

• Boolean

• Required

• No

• Default

• false

rgw enable ops log

• Description

• Enables logging of every RGW operation.

• Type

• Boolean

• Required

• No

• Default

• true

rgw enable usage log

• Description

• Enable logging of RGW’s bandwidth usage.

• Type

• Boolean

• Required

• No

• Default

• false

rgw usage log flush threshold

• Description

• Threshold to flush pending log data.

• Type

• Integer

• Required

• No

• Default

• 1024

rgw usage log tick interval

• Description

• Flush pending log data every s seconds.

• Type

• Integer

• Required

• No

• Default

• 30

rgw intent log object name

• Description

• Type

• String

• Required

• No

• Default

• %Y-%m-%d-%i-%n

rgw intent log object name utc

• Description

• Include a UTC timestamp in the intent log object name.

• Type

• Boolean

• Required

• No

• Default

• false

• 1
• there are levels >20 in some rare cases and that they are extremely verbose.