rqlite Design

Learn about the design and implementation of the database

rqlite has been in development since 2014, and its design and implementation has evolved substantially during that time. The distributed consensus system has changed, the API has improved enormously, and support for automatic clustering and node-discovery was introduced along the way.

High-level design

The diagram below shows a high-level view of a rqlite node, as it’s currently implemented. node-design

Design presentations

There have also been a series of presentations to various groups – both industry and academic.

Blog posts

The most important design articles, linked below, show how the database has evolved through the years:

You can find many other details on rqlite from the rqlite blog.

Other design details

Raft

The Raft layer always creates a file – it creates the Raft log. This log stores the set of committed SQLite commands, in the order which they were executed. This log is authoritative record of every change that has happened to the system. It may also contain some read-only queries as entries, depending on read-consistency choices. Since every node in an rqlite cluster applies the entries log in exactly the same way, this guarantees that the SQLite database is the same on every node.

SQLite

By default, the SQLite layer doesn’t create a file. Instead, it creates the database in memory. rqlite can create the SQLite database on disk, if so configured at start-time, by passing -on-disk to rqlited at startup. Regardless of whether rqlite creates a database entirely in memory, or on disk, the SQLite database is completely recreated everytime rqlited starts, using the information stored in the Raft log.

Log Compaction and Truncation

rqlite automatically performs log compaction, so that disk usage due to the log remains bounded. After a configurable number of changes rqlite snapshots the SQLite database, and truncates the Raft log. This is a technical feature of the Raft consensus system, and most users of rqlite need not be concerned with this.

Last modified March 9, 2023: Re-arrange design page (37c95c3)