Perform Maintenance on Replica Set Members

Overview

Replica sets allow a MongoDB deployment toremain available during the majority of a maintenance window.

This document outlines the basic procedure for performing maintenance oneach of the members of a replica set. Furthermore, this particularsequence strives to minimize the amount of time that theprimary is unavailable and control the impact on theentire deployment.

Use these steps as the basis for common replica set operations,particularly for procedures such as upgrading to the latestversion of MongoDB.

Procedure

For each member of a replica set, starting with a secondary member,perform the following sequence of events, ending with the primary:

  • Restart the mongod instance as a standalone.
  • Perform the task on the standalone instance.
  • Restart the mongod instance as a member of the replicaset.

Stop a secondary.

In the mongo shell, shut down the mongod instance:

  1. db.shutdownServer()

Restart the secondary as a standalone on a different port.

At the operating system shell prompt restart mongodas a standalone instance.

  • Configuration File
  • Command-line Options

If you are using a configuration file, make the followingconfiguration updates:

For example, if performing maintenance on a shard/configserver replica set member for maintenance, the updatedconfiguration file will include content like thefollowing example:

  1. net:
  2.    bindIp: localhost,<hostname(s)|ip address(es)>
  3.    port: 27218
  4. #   port: 27018
  5. #replication:
  6. #   replSetName: shardA
  7. #sharding:
  8. #   clusterRole: shardsvr
  9. setParameter:
  10.    skipShardingConfigurationChecks: true
  11.    disableLogicalSessionCacheRefresh: true

If using command-line options, make the followingconfiguration updates to restart:

For example, to restart a replica set member that is not part of a sharded cluster:

  1. mongod --port 27218 --dbpath /srv/mongodb --bind_ip localhost,<hostname(s)|ip address(es)> --setParameter disableLogicalSessionCacheRefresh=true

For example, to restart a shard/config server replica set member for maintenance:

  1. mongod --port 27218 --dbpath /srv/mongodb --bind_ip localhost,<hostname(s)|ip address(es)> --setParameter skipShardingConfigurationChecks=true --setParameter disableLogicalSessionCacheRefresh=true

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

Always start mongod with the same user, even whenrestarting a replica set member as a standalone instance.

Perform maintenance operations on the secondary.

While the member is a standalone, use the mongo shell toperform maintenance:

  1. mongo --port 27218

Important

While the member is a standalone, no writes are replicated tothis member nor are writes on this member replicated to the othermembers of the replica set.

Ensure that any writes on this standalone do not conflict withoplog writes that will be applied to the member when it rejoinsthe replica set.

Restart mongod as a member of the replica set.

After performing all maintenance tasks, use the following procedureto restart the mongod as a member of the replica seton its usual port.

From the mongo shell, shut down the standaloneserver after completing the maintenance:

  1. use admin
  2. db.shutdownServer()

Restart the mongod instance as a replica setmember with its original configuration; that is, undo theconfiguration changes made when starting as a standalone.

Tip

Be sure to remove the disableLogicalSessionCacheRefreshparameter.

For shard or config server members, be sure to remove theskipShardingConfigurationChecks parameter.

When it has started, connect the mongo shell to therestarted instance.

The secondary takes time to catch up to the primary. From the mongo shell, use thefollowing command to verify that the member has caught up from theRECOVERING state to the SECONDARY state.

  1. rs.status()

Perform maintenance on the primary last.

  • To perform maintenance on the primary after completingmaintenance tasks on all secondaries, connect amongo shell to the primary and users.stepDown() to step down the primary and allow one ofthe secondaries to be elected the new primary. Specify a 300second waiting period to prevent the member from being electedprimary again for five minutes:
  1. rs.stepDown(300)

After the primary steps down, the replica set will elect anew primary.

  • Restart mongod as a standalone instance, makingthe following configuration updates.
  • Configuration File
  • Command-line Options

If you are using a configuration file, make the followingconfiguration updates:

For example, if performing maintenance on a shard/configserver replica set member for maintenance, the updatedconfiguration file will include content like thefollowing example:

  1. net:
  2.    bindIp: localhost,<hostname(s)|ip address(es)>
  3.    port: 27218
  4. #   port: 27018
  5. #replication:
  6. #   replSetName: shardA
  7. #sharding:
  8. #   clusterRole: shardsvr
  9. setParameter:
  10.    skipShardingConfigurationChecks: true
  11.    disableLogicalSessionCacheRefresh: true

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

If using command-line options, make the followingconfiguration updates:

For example, to restart a replica set member that is not part of a sharded cluster:

  1. mongod --port 27218 --dbpath /srv/mongodb --bind_ip localhost,<hostname(s)|ip address(es)> --setParameter disableLogicalSessionCacheRefresh=true

For example, to restart a shard/config server replica set member for maintenance:

  1. mongod --port 27218 --dbpath /srv/mongodb --bind_ip localhost,<hostname(s)|ip address(es)> --setParameter skipShardingConfigurationChecks=true --setParameter disableLogicalSessionCacheRefresh=true

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

  • Perform maintentance task on the now standalone.

Important

While the member is a standalone, no writes are replicated tothis member nor are writes on this member replicated to the othermembers of the replica set.

Ensure that any writes on this standalone do not conflict withoplog writes that will be applied to the member when it rejoinsthe replica set.

  • After performing all maintenance tasks, restart themongod instance as a replica set member with itsoriginal configuration; that is, undo the configuration changes madewhen starting as a standalone.

Tip

Be sure to remove the disableLogicalSessionCacheRefreshparameter.

For shard or config server members, be sure to remove theskipShardingConfigurationChecks parameter.