Upgrade a Sharded Cluster to 4.0

Important

MongoDB 4.0 may lose data during unclean shutdowns on macOS 10.12.x and 10.13.x.

For details, see WT-4018.

Important

Before you attempt any upgrade, please familiarize yourself with thecontent of this document.

If you need guidance on upgrading to 4.0, MongoDB offers major versionupgrade services to help ensurea smooth transition without interruption to your MongoDB application.

Upgrade Recommendations and Checklists

When upgrading, consider the following:

Upgrade Version Path

To upgrade an existing MongoDB deployment to 4.0, you must berunning a 3.6-series release.

To upgrade from a version earlier than the 3.6-series, you mustsuccessively upgrade major releases until you have upgraded to3.6-series. For example, if you are running a 3.4-series, you mustupgrade first to 3.6before you can upgrade to 4.0.

Preparedness

Before beginning your upgrade, see the Compatibility Changes in MongoDB 4.0 document toensure that your applications and deployments are compatible withMongoDB 4.0. Resolve the incompatibilities in your deployment beforestarting the upgrade.

Before upgrading MongoDB, always test your application in a stagingenvironment before deploying the upgrade to your productionenvironment.

Downgrade Consideration

Once upgraded to 4.0, if you need to downgrade, we recommend downgrading to the latest patch release of 3.6.

Default Bind to Localhost

The following procedure includes the command-line option—bind_ip or the configuration optionnet.bindIp when restarting the replica set.

Starting in MongoDB 3.6, the options must be specified when the replicaset members are run on different hosts or if remote clients connect tothe deployment. Omit if all members are run on the same host and allclients are local to the host.

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

Read Concern Majority (3-Member Primary-Secondary-Arbiter Architecture)

Starting in MongoDB 3.6, MongoDB enables support for"majority" read concern by default.

For MongoDB 4.0.3+ (and 3.6.1+), you can disable read concern"majority" to prevent the storage cache pressure fromimmobilizing a deployment with a three-member replica set with aprimary-secondary-arbiter (PSA) architecture or a sharded cluster witha three-member PSA shards.

Note

Disabling "majority" read concern disables supportfor Change Streams for MongoDB 4.0 and earlier. For MongoDB4.2+, disabling read concern "majority" has no effect on changestreams availability.

Disabling "majority" does not affectmulti-document transactions; i.e. youcan specify read concern "majority" for transactionseven if read concern "majority" is disabled.

For more information, see Disable Read Concern Majority.

Change Streams Resume Tokens

MongoDB 4.0 introduces new hex-encoded string change streamresume tokens:

The resume token _data type depends on the MongoDB versions and,in some cases, the feature compatibility version (fcv) at the timeof the change stream’s opening/resumption (i.e. a change in fcvvalue does not affect the resume tokens for already opened changestreams):

MongoDB VersionFeature Compatibility VersionResume Token _data Type
MongoDB 4.0.7 and later“4.0” or “3.6”Hex-encoded string (v1)
MongoDB 4.0.6 and earlier“4.0”Hex-encoded string (v0)
MongoDB 4.0.6 and earlier“3.6”BinData
MongoDB 3.6“3.6”BinData

When upgrading from MongoDB 3.6 to MongoDB 4.0.7

When upgrading from MongoDB 3.6 to MongoDB 4.0.7 or later, theupgraded sharded cluster members continue to produce v0 tokenuntil the first mongos instance is upgraded. If aclient tries to resume change streams using the new v1 resumetoken when connected to another mongos that has notbeen updated (i.e. only accepts BinData resume tokens), the resumeoperation will fail. In such cases, the client must wait for theupgrade to complete before resuming change streams.

After upgrading, if you later decide to downgrade to MongoDB 3.6, toresume a change stream, clients can use a pre-upgrade resume token(if available) on the 3.6 deployment. Otherwise, clients will needto start a new change stream.

Prerequisites

All Members Version

To upgrade a sharded cluster to 4.0, all members of thecluster must be at least version 3.6. The upgrade processchecks all components of the cluster and will produce warnings if anycomponent is running version earlier than 3.6.

Replica Set Member State

For shards and config servers, ensure that no replica set member is inROLLBACK or RECOVERING state.

Remove Support for MONGODB-CR

Starting in version 4.0, MongoDB removes support for the deprecatedMongoDB Challenge-Response (MONGODB-CR) authentication mechanism.

If your deployment has user credentials stored in MONGODB-CRschema, you must upgrade to Salted Challenge ResponseAuthentication Mechanism (SCRAM)before youupgrade to version 4.0. For information on upgrading to SCRAM, seeUpgrade to SCRAM.

See also

Compatibility Changes in MongoDB 4.0

Remove pv0 for Replica Sets

Starting in version 4.0, MongoDB removes the deprecated replica setprotocol version 0 pv0.

Before upgrading to MongoDB 4.0, you must upgrade to pv1.

To upgrade to pv1, connect a mongo shell to thereplica set primary and perform the following sequence of operations:

  1. cfg = rs.conf();
  2. cfg.protocolVersion=1;
  3. rs.reconfig(cfg);

To reduce the likelihood of w:1 rollbacks,you can also reconfigure the replica set to a highersettings.catchUpTimeoutMillis setting.

For more information on pv1, seeReplica Set Protocol Version.

Remove Master-Slave Replication

MongoDB 4.0 removes support for the deprecated master-slavereplication. Before you can upgrade to MongoDB 4.0, if your deploymentuses master-slave replication, you must upgrade to a replica set.

To convert your master-slave replication, seeConvert a Master-Slave Deployment to a Replica Set.

Remove Support for $isolated

MongoDB drops support for the $isolated operator. If you have anexisting partial index that includes the $isolated operator or aview that includes a $isolated operator, recreate the index orview without the operator in the definition before upgrading.

Feature Compatibility Version

The 3.6 sharded cluster must havefeatureCompatibilityVersion set to 3.6.

To ensure that all members of the sharded cluster havefeatureCompatibilityVersion set to 3.6, connect to eachshard replica set member and each config server replica set memberand check the featureCompatibilityVersion:

Tip

For a sharded cluster that has access control enabled, to run thefollowing command against a shard replica set member, you mustconnect to the member as a shard local user.

  1. db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

All members should return a result that includes"featureCompatibilityVersion" : { "version" : "3.6" }.

To set or update featureCompatibilityVersion, run thefollowing command on the mongos:

  1. db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

For more information, seesetFeatureCompatibilityVersion.

Back up the config Database

Optional but Recommended. As a precaution, take a backup of theconfig database before upgrading the sharded cluster.

Download 4.0 Binaries

Use Package Manager

If you installed MongoDB from the MongoDB apt, yum, dnf, orzypper repositories, you should upgrade to 4.0 using your packagemanager.

Follow the appropriate 4.0 installation instructions for your Linux system. Thiswill involve adding a repository for the new release, then performingthe actual upgrade process.

Download 4.0 Binaries Manually

If you have not installed MongoDB using a package manager, you canmanually download the MongoDB binaries from the MongoDB DownloadCenter.

See 4.0 installation instructions for more information.

Upgrade Process

Disable the Balancer.

Connect a mongo shell to a mongos instance inthe sharded cluster, and run sh.stopBalancer() todisable the balancer:

  1. sh.stopBalancer()

Note

If a migration is in progress, the system will complete thein-progress migration before stopping the balancer. You can runsh.isBalancerRunning() to check the balancer’s currentstate.

To verify that the balancer is disabled, runsh.getBalancerState(), which returns false if the balanceris disabled:

  1. sh.getBalancerState()

For more information on disabling the balancer, seeDisable the Balancer.

Upgrade the config servers.

  • Upgrade the secondarymembers of the replica set one at a time:

    • Shut down the secondary mongod instance and replacethe 3.6 binary with the 4.0 binary.

    • Start the 4.0 binary with the —configsvr,—replSet, and —port.Include any other options as used by the deployment.

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.
  1. mongod --configsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

If using a configuration file, update the file tospecify sharding.clusterRole: configsvr, replication.replSetName,net.port, and net.bindIp,then start the 4.0 binary:

  1. sharding:
  2.    clusterRole: configsvr
  3. replication:
  4.    replSetName: <string>
  5. net:
  6.    port: <port>
  7.    bindIp: localhost,<hostname(s)|ip address(es)>
  8. storage:
  9.    dbpath: <path>

Include any other settings as appropriate for your deployment.

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.
  • Wait for the member to recover to SECONDARY state beforeupgrading the next secondary member. To check the member’s state,issue rs.status() in the mongo shell.

Repeat for each secondary member.

  • Step down the replica set primary.

    • Connect a mongo shell to the primary and users.stepDown() to step down the primary and force anelection of a new primary:
  1. rs.stepDown()

  • When rs.status() shows that the primary has steppeddown and another member has assumed PRIMARY state, shut downthe stepped-down primary and replace the mongod binarywith the 4.0 binary.

  • Start the 4.0 binary with the —configsvr, —replSet,—port, and —bind_ip options. Include any optional command lineoptions used by the previous deployment:

  1. mongod --configsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.

If using a configuration file, update the file tospecify sharding.clusterRole: configsvr, replication.replSetName,net.port, and net.bindIp,then start the 4.0 binary:

  1. sharding:
  2.    clusterRole: configsvr
  3. replication:
  4.    replSetName: <string>
  5. net:
  6.    port: <port>
  7.    bindIp: localhost,<hostname(s)|ip address(es)>
  8. storage:
  9.    dbpath: <path>

Include any other configuration as appropriate for your deployment.

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.

Upgrade the shards.

Upgrade the shards one at a time.

For each shard replica set:

  • Upgrade the secondarymembers of the replica set one at a time:

    • Shut down the mongod instance and replace the 3.6binary with the 4.0 binary.

    • Start the 4.0 binary with the —shardsvr, —replSet,—port, and —bind_ip options. Include any additional command lineoptions as appropriate for your deployment:

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.
  1. mongod --shardsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

If using a configuration file, update the file toinclude sharding.clusterRole: shardsvr,replication.replSetName, net.port, andnet.bindIp, then start the 4.0 binary:

  1. sharding:
  2.    clusterRole: shardsvr
  3. replication:
  4.    replSetName: <string>
  5. net:
  6.    port: <port>
  7.    bindIp: localhost,<hostname(s)|ip address(es)>
  8. storage:
  9.    dbpath: <path>

Include any other configuration as appropriate for your deployment.

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.
  • Wait for the member to recover to SECONDARY state beforeupgrading the next secondary member. To check the member’sstate, you can issue rs.status() in themongo shell.

Repeat for each secondary member.

  • Step down the replica set primary.

Connect a mongo shell to the primary and users.stepDown() to step down the primary and force anelection of a new primary:

  1. rs.stepDown()

  • When rs.status()shows that the primary has stepped down and another memberhas assumed PRIMARY state, upgrade the stepped-down primary:

    • Shut down the stepped-down primary and replace themongod binary with the 4.0 binary.

    • Start the 4.0 binary with the —shardsvr, —replSet,—port, and —bind_ip options. Include any additional command lineoptions as appropriate for your deployment:

  1. mongod --shardsvr --replSet <replSetName> --port <port> --dbpath <path> --bind_ip localhost,<hostname(s)|ip address(es)>

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.

If using a configuration file, update the file tospecify sharding.clusterRole: shardsvr, replication.replSetName,net.port, and net.bindIp, then start the4.0 binary:

  1. sharding:
  2.    clusterRole: shardsvr
  3. replication:
  4.    replSetName: <string>
  5. net:
  6.    port: <port>
  7.    bindIp: localhost,<hostname(s)|ip address(es)>
  8. storage:
  9.    dbpath: <path>

Include any other configuration as appropriate for your deployment.

Note

  1. - Starting in MongoDB 4.0, you cannot specify [<code>--nojournal</code>]($1cde9863b9db90a4.md#cmdoption-mongod-nojournal) option or [<code>storage.journal.enabled:

false]($1e2c675a7cc55c0f.md#storage.journal.enabled) for replica set members that use theWiredTiger storage engine.

  1. - Starting in MongoDB 4.0, you cannot use [<code>--noIndexBuildRetry</code>]($1cde9863b9db90a4.md#cmdoption-mongod-noindexbuildretry) or [<code>storage.indexBuildRetry</code>]($1e2c675a7cc55c0f.md#storage.indexBuildRetry) for a[<code>mongod</code>]($1cde9863b9db90a4.md#bin.mongod) instance that is part of a replica set.

Upgrade the mongos instances.

Replace each mongos instance with the 4.0 binaryand restart. Include any other configuration as appropriate for yourdeployment.

Note

The —bind_ip option must be specified whenthe sharded cluster members are run on different hosts or ifremote clients connect to the sharded cluster. For more information, seeLocalhost Binding Compatibility Changes.

  1. mongos --configdb csReplSet/<rsconfigsver1:port1>,<rsconfigsver2:port2>,<rsconfigsver3:port3> --bind_ip localhost,<hostname(s)|ip address(es)>

Re-enable the balancer.

Using a 4.0 mongo shell, connect to amongos in the cluster and runsh.setBalancerState() to re-enable the balancer:

  1. sh.setBalancerState(true)

The 3.6 and earlier mongo shell is notcompatible with 4.0 clusters.

For more information about re-enabling the balancer, seeEnable the Balancer.

Enable backwards-incompatible 4.0 features.

At this point, you can run the 4.0 binaries without the4.0 features that are incompatible with 3.6.

To enable these 4.0 features, set the feature compatibilityversion (FCV) to 4.0.

Tip

Enabling these backwards-incompatible features can complicate thedowngrade process since you must remove any persistedbackwards-incompatible features before you downgrade.

It is recommended that after upgrading, you allow your deployment torun without enabling these features for a burn-in period to ensurethe likelihood of downgrade is minimal. When you are confident thatthe likelihood of downgrade is minimal, enable these features.

On a mongos instance, run thesetFeatureCompatibilityVersion command in the admindatabase:

  1. db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )

This command must perform writes to an internal systemcollection. If for any reason the command does not completesuccessfully, you can safely retry the command on themongos as the operation is idempotent.

Restart mongos instances.

After changing the featureCompatibilityVersion, allmongos instances need to be restarted to pick up thechanges in the causal consistency behavior.

Additional Upgrade Procedures