In-Memory Storage Engine

Changed in version 3.2.6.

Starting in MongoDB Enterprise version 3.2.6, the in-memory storageengine is part of general availability (GA) in the 64-bit builds. Otherthan some metadata and diagnostic data, the in-memory storage enginedoes not maintain any on-disk data, including configuration data,indexes, user credentials, etc.

By avoiding disk I/O, the in-memory storage engine allows for morepredictable latency of database operations.

Specify In-Memory Storage Engine

To select the in-memory storage engine, specify:

  • inMemory for the —storageEngine option, or thestorage.engine setting if using a configuration file.
  • —dbpath, or storage.dbPath if using a configurationfile. Although the in-memory storage engine does not write data tothe filesystem, it maintains in the —dbpath small metadata filesand diagnostic data as well temporary files for building largeindexes.

For example, from the command line:

  1. mongod --storageEngine inMemory --dbpath <path>

Or, if using the YAML configuration file format:

  1. storage:
  2.    engine: inMemory
  3.    dbPath: <path>

See inMemory Options for configuration options specific tothis storage engine. Most mongod configuration options areavailable for use with in-memory storage engine except for thoseoptions that are related to data persistence, such as journaling orencryption at rest configuration.

Warning

The in-memory storage engine does not persist data after process shutdown.

Concurrency

The in-memory storage engine uses document-level concurrency control for writeoperations. As a result, multiple clients can modify differentdocuments of a collection at the same time.

Memory Use

In-memory storage engine requires that all its data (including indexes,oplog if mongod instance is part of a replica set, etc.) mustfit into the specified —inMemorySizeGB command-line optionor storage.inMemory.engineConfig.inMemorySizeGB setting inthe YAML configuration file.

By default, the in-memory storage engine uses 50% of physical RAM minus1 GB.

If a write operation would cause the data to exceed the specifiedmemory size, MongoDB returns with the error:

  1. "WT_CACHE_FULL: operation would overflow cache"

To specify a new size, use thestorage.inMemory.engineConfig.inMemorySizeGB setting in theYAML configuration file format:

  1. storage:
  2.    engine: inMemory
  3.    dbPath: <path>
  4.    inMemory:
  5.       engineConfig:
  6.          inMemorySizeGB: <newSize>

Or use the command-line option —inMemorySizeGB:

  1. mongod --storageEngine inMemory --dbpath <path> --inMemorySizeGB <newSize>

Durability

The in-memory storage engine is non-persistent and does not write datato a persistent storage. Non-persisted data includesapplication data and system data, such as users, permissions, indexes,replica set configuration, sharded cluster configuration, etc.

As such, the concept of journal or waiting for data to becomedurable does not apply to the in-memory storage engine.

If any voting member of a replica set uses the in-memorystorage engine, you must setwriteConcernMajorityJournalDefault to false.

Note

Starting in version 4.2 (and 4.0.13 and 3.6.14 ), if a replica setmember uses the in-memory storage engine(voting or non-voting) but the replica set haswriteConcernMajorityJournalDefault set to true, thereplica set member logs a startup warning.

With writeConcernMajorityJournalDefault set to false,MongoDB does not wait for w: "majority"writes to be written to the on-disk journal before acknowledging thewrites. As such, majority write operations couldpossibly roll back in the event of a transient loss (e.g. crash andrestart) of a majority of nodes in a given replica set.

Write operations that specify a write concern journaled are acknowledged immediately. When an mongod instanceshuts down, either as result of the shutdown command ordue to a system error, recovery of in-memory data is impossible.

Transactions

Starting in MongoDB 4.2, transactions are supported on replica setsand sharded clusters where:

  • the primary uses the WiredTiger storageengine, and
  • the secondary members use either the WiredTiger storage engine or thein-memory storage engines.

In MongoDB 4.0, only replica sets using the WiredTiger storageengine supported transactions.

Note

You cannot run transactions on a sharded cluster that has a shardwith writeConcernMajorityJournalDefault set tofalse, such as a shard with a voting member that uses thein-memory storage engine.

Deployment Architectures

In addition to running as standalones, mongod instances thatuse in-memory storage engine can run as part of a replica set or partof a sharded cluster.

Replica Set

You can deploy mongod instances that use in-memory storageengine as part of a replica set. For example, as part of a three-memberreplica set, you could have:

With this deployment model, only the mongod instancesrunning with the in-memory storage engine can become the primary.Clients connect only to the in-memory storage engine mongodinstances. Even if both mongod instances running in-memorystorage engine crash and restart, they can sync from the member runningWiredTiger. The hidden mongod instance running withWiredTiger persists the data to disk, including the user data, indexes,and replication configuration information.

Note

In-memory storage engine requires that all its data (including oplogif mongod is part of replica set, etc.) fit into thespecified —inMemorySizeGB command-line option orstorage.inMemory.engineConfig.inMemorySizeGB setting. SeeMemory Use.

Sharded Cluster

You can deploy mongod instances that use in-memory storageengine as part of a sharded cluster. For example, in a sharded cluster,you could have one shard that has consists of the following replica set:

To this shard, add the tag inmem. Forexample, if this shard has the name shardC, connect to themongos and run sh.addShardTag().

For example,

  1. sh.addShardTag("shardC", "inmem")

To the other shards, add a separate tag persisted .

  1. sh.addShardTag("shardA", "persisted")
  2. sh.addShardTag("shardB", "persisted")

For each sharded collection that should reside on the inmem shard,assign to the entire chunk range the taginmem:

  1. sh.addTagRange("test.analytics", { shardKey: MinKey }, { shardKey: MaxKey }, "inmem")

For each sharded collection that should reside across thepersisted shards, assign to the entire chunk range the tagpersisted:

  1. sh.addTagRange("salesdb.orders", { shardKey: MinKey }, { shardKey: MaxKey }, "persisted")

For the inmem shard, create a database or move the database.