Logging

Apache Druid processes will emit logs that are useful for debugging to log files. These processes also emit periodic metrics about their state. Metric info logs can be disabled with -Ddruid.emitter.logging.logLevel=debug.

Druid uses log4j2 for logging. The default configuration file log4j2.xml ships with Druid under conf/druid/{config}/_common/log4j2.xml .

By default, Druid uses RollingRandomAccessFile for rollover daily, and keeps log files up to 7 days. If that’s not suitable in your case, you could modify the log4j2.xml to meet your need.

An example log4j2.xml file is shown below:

  1. <?xml version="1.0" encoding="UTF-8" ?>
  2. <Configuration status="WARN">
  3.   <Console name="Console" target="SYSTEM_OUT">
  4.     <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
  5.   </Console>
  7.   <RollingRandomAccessFile name="FileAppender"
  8.                            fileName="${sys:druid.log.path}/${sys:druid.node.type}.log"
  9.                            filePattern="${sys:druid.log.path}/${sys:druid.node.type}.%d{yyyyMMdd}.log">
  10.     <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
  11.     <Policies>
  12.         <TimeBasedTriggeringPolicy interval="1" modulate="true"/>
  13.     </Policies>
  14.     <DefaultRolloverStrategy>
  15.       <Delete basePath="${sys:druid.log.path}/" maxDepth="1">
  16.         <IfFileName glob="*.log" />
  17.         <IfLastModified age="7d" />
  18.       </Delete>
  19.     </DefaultRolloverStrategy>
  20.   </RollingRandomAccessFile>
  21.   <Loggers>
  22.     <Root level="info">
  23.       <AppenderRef ref="FileAppender"/>
  24.     </Root>
  26.     <!-- Uncomment to enable logging of all HTTP requests
  27.     <Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG">
  28.         <AppenderRef ref="FileAppender"/>
  29.     </Logger>
  30.     -->
  31.   </Loggers>
  32. </Configuration>

NOTE: Although the log4j configuration file is shared with Druid’s task peon processes, the appenders in this file DO NOT take effect for peon processes, which always output logs to standard output. Middle Managers redirect task logs from standard output to long-term storage.

However, log level settings do take effect for these task peon processes, which means you can still configure loggers at different logging level for task logs using log4j2.xml.

Log directory

The included log4j2.xml configuration for Druid and ZooKeeper will output logs to the log directory at the root of the distribution.

If you want to change the log directory, set the environment variable DRUID_LOG_DIR to the right directory before you start Druid.

All-in-one start commands

If you use one of the all-in-one start commands, such as bin/start-micro-quickstart, then in the default configuration each service has two kind of log files. The main log file (for example, log/historical.log) is written by log4j2 and is rotated periodically.

The secondary log file (for example, log/historical.stdout.log) contains anything that is written by the component directly to standard output or standard error without going through log4j2. This consists mainly of messages from the Java runtime itself. This file is not rotated, but it is generally small due to the low volume of messages. If necessary, you can truncate it using the Linux command truncate --size 0 log/historical.stdout.log.

Avoid reflective access warnings in logs

On Java 11, you may see warnings like this in log files:

  1. WARNING: An illegal reflective access operation has occurred
  2. WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  3. WARNING: All illegal access operations will be denied in a future release

These messages do not cause harm, but you can avoid them by adding the following lines to your jvm.config files. These lines are not part of the default JVM configs that ship with Druid, because Java 8 will not recognize these options and will fail to start up.

  1. --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
  2. --add-exports=java.base/jdk.internal.perf=ALL-UNNAMED
  3. --add-opens=java.base/java.lang=ALL-UNNAMED
  4. --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED

My logs are really chatty, can I set them to asynchronously write?

Yes, using a log4j2.xml similar to the following causes some of the more chatty classes to write asynchronously:

  1. <?xml version="1.0" encoding="UTF-8" ?>
  2. <Configuration status="WARN">
  3.   <Appenders>
  4.     <Console name="Console" target="SYSTEM_OUT">
  5.       <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
  6.     </Console>
  7.   </Appenders>
  9. <Loggers>
  10.     <!-- AsyncLogger instead of Logger -->
  11.     <AsyncLogger name="org.apache.druid.curator.inventory.CuratorInventoryManager" level="debug" additivity="false">
  12.       <AppenderRef ref="Console"/>
  13.     </AsyncLogger>
  14.     <AsyncLogger name="org.apache.druid.client.BatchServerInventoryView" level="debug" additivity="false">
  15.       <AppenderRef ref="Console"/>
  16.     </AsyncLogger>
  17.     <!-- Make extra sure nobody adds logs in a bad way that can hurt performance -->
  18.     <AsyncLogger name="org.apache.druid.client.ServerInventoryView" level="debug" additivity="false">
  19.       <AppenderRef ref="Console"/>
  20.     </AsyncLogger>
  21.     <AsyncLogger name ="org.apache.druid.java.util.http.client.pool.ChannelResourceFactory" level="info" additivity="false">
  22.       <AppenderRef ref="Console"/>
  23.     </AsyncLogger>
  24.     <Root level="info">
  25.       <AppenderRef ref="Console"/>
  26.     </Root>
  27.   </Loggers>
  28. </Configuration>