RocksDB provides a list of options for users to hint how I/Os should be executed.
RocksDB’s data files are usually generated in an appending way. File system may choose buffer the write until the dirty pages hit a threshold and write out all of those pages all together. This can create a burst of write I/O and cause the online I/Os to wait too long and cause long query latency. Rather, you can ask RocksDB to periodically hint OS to write out outstanding dirty pages by setting
options.bytes_per_sync for SST files and
options.wal_bytes_per_sync for WAL files. Underlying it calls sync_file_range() on Linux every time a file is appended for such size. The most recent pages are not included in the range sync.
You can control the total rate RocksDB writes to data files through
options.rate_limiter, in order to reserve enough I/O bandwidth to online queries. See Rate Limiter for details.
When appending a file, RocksDB has internal buffering of files before writing to the file system, unless an explicit fsync is needed. The max size of this buffer can be controlled by
options.writable_file_max_buffer_size. Tuning this parameter is more critical in [[Direct IO] mode or to a file system without page cache. With non-direct I/O mode, enlarging this buffer only reduces number of write() system calls and is unlikely to change the I/O behavior, so unless this is what you want, it may be desirable to keep the default value 0 to save the memory.
Deletion of obsolete DB files can be rate limited by configuring the delete scheduler. This is especially useful on flash devices to limit read latency spikes due to a burst of deletions. See Delete Scheduler for details.
While opening an SST file for reads, users can decide whether RocksDB will call fadvise with FADV_RANDOM, by setting
options.advise_random_on_open = true (default). If the value is
false, no fadvise will be called while opening a file. Setting the option to be
true usually works well if the dominating queries are either Get() or iterating a very short range, because read-ahead is not helpful in these cases anyway. Otherwise,
options.advise_random_on_open = false can usually improve performance to hint the file system to do underlying read-ahead.
Unfortunately, there isn’t a good setting for mixed workload. There is an ongoing project to address this, by doing read-ahead for iterators inside RocksDB.
Compaction inputs are special. They are long sequential reads, so applying the same fadvise hint as user reads is usually not optimal. Also, usually, compaction input files are often going to be deleted soon, though there is no guarantee. RocksDB provides multiple ways to deal with that:
RocksDB will call fadvise to any compaction input file according to
options.access_hint_on_compaction_start. This can override the fadvise random setting since a file is picked as a compaction input.
options.new_table_reader_for_compaction_inputs = true, RocksDB will use different file descriptors for compaction inputs. This can avoid the mixture of fadvise setting for normal data files and compaction inputs. The limitation of the setting is that, RocksDB does not just create a new file descriptor, but read index, filter and other meta blocks again and store them in memory, so that it takes extra I/O and use more memory.
You can do its own readahead following
options.compaction_readahead_size if it is not 0.
options.new_table_reader_for_compaction_inputs is automatically switched to
true if the option is set. This setting can allow users to keep
It is critical to set the option if Direct IO is on or the file system doesn’t support readahead.
Rather than control the I/O through file system hints shown above, you can enable direct I/O in RocksDB to allow RocksDB to directly control I/O, using option
use_direct_io_for_flush_and_compaction. If direct I/O is enabled, some or all of the options introduced above will not be applicable. See more details in Direct IO.
options.allow_mmap_writes make RocksDB mmap the whole data file while doing read or write, respectively. The benefit of the approach is to reduce the file system calls doing pread() and write(), and in many cases, reduce the memory copying too.
options.allow_mmap_reads can usually significantly improve performance if the DB is run on ramfs. They can be used on file systems backed by block device too. However, based on our previous experience, file systems aren’t usually doing a perfect job maintaining this kind of memory mapping, and some times cause slow queries. In this case, we advise you try out this option only when necessary and with caution.
Iterator destruction and cleanup on column family destruction by default will try to delete obsolete files in the context of the thread calling the destructor, subject to deletion rate limits. This can result in an unexpected long latency in completing the operation. To avoid the long latency and defer the deletion of obsolete files to background threads, the following options are provided -
ReadOptions::background_purge_on_iterator_cleanup- This option, when set in the call to
DB::NewIterator(), will schedule the deletion of obsolete files in a background thread on iterator destruction.
DBOptions::avoid_unnecessary_blocking_io- This option is a DB wide option. When set, both iterator destructor and
ColumnFamilyHandledestructors will schedule obsolete file deletion in a background thread.