Use of the cluster log

(Note: none of this applies to the local “dout” logging. This is aboutthe cluster log that we send through the mon daemons)

Severity

Use ERR for situations where the cluster cannot do its job for some reason.For example: we tried to do a write, but it returned an error, or we triedto read something, but it’s corrupt so we can’t, or we scrubbed a PG butthe data was inconsistent so we can’t recover.

Use WRN for incidents that the cluster can handle, but have some abnormal/negativeaspect, such as a temporary degradation of service, or an unexpected internalvalue. For example, a metadata error that can be auto-fixed, or a slow operation.

Use INFO for ordinary cluster operations that do not indicate a fault inCeph. It is especially important that INFO level messages are clearlyworded and do not cause confusion or alarm.

Frequency

It is important that messages of all severities are not excessivelyfrequent. Consumers may be using a rotating log buffer that containsmessages of all severities, so even DEBUG messages could interferewith proper display of the latest INFO messages if the DEBUG messagesare too frequent.

Remember that if you have a bad state (as opposed to event), that iswhat health checks are for – do not spam the cluster log to indicatea continuing unhealthy state.

Do not emit cluster log messages for events that scale withthe number of clients or level of activity on the system, or forevents that occur regularly in normal operation. For example, itwould be inappropriate to emit a INFO message about everynew client that connects (scales with #clients), or to emit and INFOmessage about every CephFS subtree migration (occurs regularly).

Language and formatting

• (Note: these guidelines matter much less for DEBUG-level messages than
• for INFO and above. Concentrate your efforts on making INFO/WRN/ERRmessages as readable as possible.)

Use the passive voice. For example, use “Object xyz could not be read”, ratherthan “I could not read the object xyz”.

Print long/big identifiers, such as inode numbers, as hex, prefixedwith an 0x so that the user can tell it is hex. We do this becausethe 0x makes it unambiguous (no equivalent for decimal), and becausethe hex form is more likely to fit on the screen.

Print size quantities as a human readable MB/GB/etc, including the unitat the end of the number. Exception: if you are specifying an offset,where precision is essential to the meaning, then you can specifythe value in bytes (but print it as hex).

Make a good faith effort to fit your message on a single line. It doesnot have to be guaranteed, but it should at least usually bethe case. That means, generally, no printing of lists unless thereare only a few items in the list.

Use nouns that are meaningful to the user, and defined in thedocumentation. Common acronyms are OK – don’t waste screen spacetyping “Rados Object Gateway” instead of RGW. Do not use internalclass names like “MDCache” or “Objecter”. It is okay to mentioninternal structures if they are the direct subject of the message,for example in a corruption, but use plain english.Example: instead of “Objecter requests” say “OSD client requests”Example: it is okay to mention internal structure in the contextof “Corrupt session table” (but don’t say “Corrupt SessionTable”)

Where possible, describe the consequence for system availability, ratherthan only describing the underlying state. For example, rather thansaying “MDS myfs.0 is replaying”, say that “myfs is degraded, waitingfor myfs.0 to finish starting”.

While common acronyms are fine, don’t randomly truncate words. It’s not“dir ino”, it’s “directory inode”.

If you’re logging something that “should never happen”, i.e. a situationwhere it would be an assertion, but we’re helpfully not crashing, thenmake that clear in the language – this is probably not a situationthat the user can remediate themselves.

Avoid UNIX/programmer jargon. Instead of “errno”, just say “error” (orpreferably give something more descriptive than the number!)

Do not mention cluster map epochs unless they are essential tothe meaning of the message. For example, “OSDMap epoch 123 is corrupt”would be okay (the epoch is the point of the message), but saying “OSD123 is down in OSDMap epoch 456” would not be (the osdmap and epochconcepts are an implementation detail, the down-ness of the OSDis the real message). Feel free to send additional detail tothe daemon’s local log (via dout/derr).

If you log a problem that may go away in the future, make sure youalso log when it goes away. Whatever priority you logged the originalmessage at, log the “going away” message at INFO.