Replace a Config Server

Important

The following procedure applies to 4.2 config servers. Forearlier versions of MongoDB, refer to the corresponding version ofthe MongoDB Manual.

Overview

If the config server replica set becomes read only, i.e. does nothave a primary, the sharded cluster cannot support operations that changethe cluster metadata, such as chunk splits and migrations. Although nochunks can be split or migrated, applications will be able to write datato the sharded cluster.

If one of the config servers is unavailable or inoperable, repair orreplace it as soon as possible. The following procedure replaces a memberof a config server replica set with a newmember.

The tutorial is specific to MongoDB 4.2. For earlier versions ofMongoDB, refer to the corresponding version of the MongoDB Manual.

Considerations

The following restrictions apply to a replica set configuration when usedfor config servers:

  • Must have zero arbiters.
  • Must have no delayed members.
  • Must build indexes (i.e. no member should havebuildIndexes setting set tofalse).

Procedure

Start the replacement config server.

Start a mongod instance, specifying the —configsvr,—replSet, —bind_ip options, and other options asappropriate to your deployment.

Warning

Before binding to a non-localhost (e.g. publicly accessible)IP address, ensure you have secured your cluster from unauthorizedaccess. For a complete list of security recommendations, seeSecurity Checklist. At minimum, considerenabling authentication andhardening network infrastructure.

  1. mongod --configsvr --replSet <replicaSetName> --bind_ip localhost,<hostname(s)|ip address(es)>

Add the new config server to the replica set.

Connect a mongo shell to the primary of the config serverreplica set and use rs.add() to add the new member.

Tip

When a newly added secondary has its votes andpriority settings greater than zero, duringits initial sync, the secondary still counts as a voting member eventhough it cannot serve reads nor become primary because its data isnot yet consistent.

This can lead to a case where a majority of the voting members areonline but no primary can be elected. To avoid such situations,consider adding the new secondary initially withpriority :0 and votes :0. Then, once the member has transitioned intoSECONDARY state, use rs.reconfig() to update itspriority and votes.

  1. rs.add( { host: "<hostnameNew>:<portNew>", priority: 0, votes: 0 } )

The initial sync process copies all the data from one member of theconfig server replica set to the new member without restarting.

mongos instances automatically recognize the change in theconfig server replica set members without restarting.

Update the newly added config server’s votes and priority settings.

  • Ensure that the new member has reached SECONDARYstate. To check the state of the replica set members, runrs.status():
  1. rs.status()
  • Reconfigure the replica set to update the votes and priority ofthe new member:
  1. var cfg = rs.conf();
  2.  
  3. cfg.members[n].priority = 1;  // Substitute the correct array index for the new member
  4. cfg.members[n].votes = 1;     // Substitute the correct array index for the new member
  5.  
  6. rs.reconfig(cfg)

where n is the array index of the new member in themembers array.

Warning

  • The rs.reconfig() shell method can force the currentprimary to step down, which causes an election. When the primary steps down, themongod closes all client connections. While thistypically takes 10-20 seconds, try to make these changes duringscheduled maintenance periods.
  • Avoid reconfiguring replica sets that contain members of differentMongoDB versions as validation rules may differ across MongoDB versions.

Shut down the member to replace.

If replacing the primary member, step down the primary first beforeshutting down.

Remove the member to replace from the config server replica set.

Upon completion of initial sync of the replacement config server,from a mongo shell connected to the primary, users.remove() to remove the old member.

  1. rs.remove("<hostnameOld>:<portOld>")

mongos instances automatically recognize the change in theconfig server replica set members without restarting.

If necessary, update mongos configuration or DNS entry.

With replica set config servers, the mongos instances specifyin the —configdb or sharding.configDB setting the configserver replica set name and at least one of the replica set members.

As such, if the mongos instance does not specify theremoved replica set member in the —configdb orsharding.configDB setting, no further action is necessary.

If, however, a mongos instance specified the removedmember in the —configdb or configDBsetting, either:

  • Update the setting for the next time you restart themongos, or
  • Modify the DNS entry that points to the system that provided the oldconfig server, so that the same hostname points to the newconfig server.