Manage Sharded Cluster Balancer

Manage Sharded Cluster Balancer

Changed in version 3.4: The balancer process has moved from the mongos instancesto the primary member of the config server replica set.

This page describes common administrative procedures relatedto balancing. For an introduction to balancing, seeSharded Cluster Balancer. For lower level information on balancing, seeCluster Balancer.

Important

Use the version of the mongo shell that corresponds tothe version of the sharded cluster. For example, do not use a 3.2 orearlier version of mongo shell against the 3.4 shardedcluster.

Check the Balancer State

sh.getBalancerState() checks if the balancer is enabled (i.e. that thebalancer is permitted to run). sh.getBalancerState() does not check if the balanceris actively balancing chunks.

To see if the balancer is enabled in your sharded cluster,issue the following command, which returns a boolean:

sh.getBalancerState()

New in version 3.0.0: You can also see if the balancer is enabled usingsh.status(). Thecurrently-enabled field indicates whetherthe balancer is enabled, while thecurrently-running field indicates ifthe balancer is currently running.

Check if Balancer is Running

To see if the balancer process is active in your cluster:

Important

Use the version of the mongo shell that corresponds tothe version of the sharded cluster. For example, do not use a 3.2 orearlier version of mongo shell against the 3.4 shardedcluster.

Connect to any mongos in the cluster using themongo shell.
Use the following operation to determine if the balancer is running:

sh.isBalancerRunning()

Configure Default Chunk Size

The default chunk size for a sharded cluster is 64 megabytes. In mostsituations, the default size is appropriate for splitting and migratingchunks. For information on how chunk size affects deployments, seedetails, see Chunk Size.

Changing the default chunk size affects chunks that are processes duringmigrations and auto-splits but does not retroactively affect all chunks.

To configure default chunk size, seeModify Chunk Size in a Sharded Cluster.

Schedule the Balancing Window

In some situations, particularly when your data set grows slowly and amigration can impact performance, it is useful to ensurethat the balancer is active only at certain times. The followingprocedure specifies the activeWindow,which is the timeframe during which the balancer willbe able to migrate chunks:

Connect to mongos using the mongo shell.

You can connect to any mongos in the cluster.

Switch to the Config Database.

Issue the following command to switch to the config database.

use config

Ensure that the balancer is not stopped.

The balancer will not activate in the stopped state.To ensure that the balanceris not stopped, use sh.startBalancer(),as in the following:

sh.startBalancer()

The balancer will not start if you are outsideof the activeWindow timeframe.

Starting in MongoDB 4.2, sh.startBalancer() also enablesauto-splitting for the sharded cluster.

Modify the balancer’s window.

Set the activeWindow using update(),as in the following:

db.settings.update(
   { _id: "balancer" },
   { $set: { activeWindow : { start : "<start-time>", stop : "<stop-time>" } } },
   { upsert: true }
)

Replace <start-time> and <end-time> with time values usingtwo digit hour and minute values (i.e. HH:MM) that specify thebeginning and end boundaries of the balancing window.

For HH values, use hour values ranging from 00 - 23.
For MM value, use minute values ranging from 00 - 59.

MongoDB evaluates the start and stop times relative to the timezone of the member which is serving as a primary in the configserver replica set.

Note

The balancer window must be sufficient to complete the migrationof all data inserted during the day.

As data insert rates can change based on activity and usagepatterns, it is important to ensure that the balancing window youselect will be sufficient to support the needs of your deployment.

Do not use the sh.startBalancer() method when you have set anactiveWindow.

Remove a Balancing Window Schedule

If you have set the balancing window and wish to remove the scheduleso that the balancer is always running, use $unset to clearthe activeWindow, as in the following:

use config
db.settings.update({ _id : "balancer" }, { $unset : { activeWindow : true } })

Disable the Balancer

By default, the balancer may run at any time and only moves chunks asneeded. To disable the balancer for a short period of time and preventall migration, use the following procedure:

Connect to any mongos in the cluster using themongo shell.
Issue the following operation to disable the balancer:

sh.stopBalancer()

If a migration is in progress, the system will complete thein-progress migration before stopping.

Starting in MongoDB 4.2, sh.stopBalancer() also disablesauto-splitting for the sharded cluster.

To verify that the balancer will not start, issue the following command,which returns false if the balancer is disabled:

sh.getBalancerState()

Optionally, to verify no migrations are in progress after disabling,issue the following operation in the mongo shell:

use config
while( sh.isBalancerRunning() ) {
          print("waiting...");
          sleep(1000);
}

Note

To disable the balancer from a driver,use the balancerStop command against the admin database,as in the following:

db.adminCommand( { balancerStop: 1 } )

Enable the Balancer

Use this procedure if you have disabled the balancer and are ready tore-enable it:

Connect to any mongos in the cluster using themongo shell.
Issue one of the following operations to enable the balancer:

From the mongo shell, issue:

sh.startBalancer()

Note

To enable the balancer from a driver, use the balancerStartcommand against the admin database, as in the following:

db.adminCommand( { balancerStart: 1 } )

Starting in MongoDB 4.2, sh.startBalancer() also enablesauto-splitting for the sharded cluster.

Disable Balancing During Backups

If MongoDB migrates a chunk during a backup, you can end with an inconsistent snapshotof your sharded cluster. Never run a backup while the balancer isactive. To ensure that the balancer is inactive during your backupoperation:

Set the balancing windowso that the balancer is inactive during the backup. Ensure that thebackup can complete while you have the balancer disabled.
manually disable the balancerfor the duration of the backup procedure.

If you turn the balancer off while it is in the middle of a balancing round,the shut down is not instantaneous. The balancer completes the chunkmove in-progress and then ceases all further balancing rounds.

Before starting a backup operation, confirm that the balancer is notactive. You can use the following command to determine if the balanceris active:

!sh.getBalancerState() && !sh.isBalancerRunning()

When the backup procedure is complete you can reactivatethe balancer process.

Disable Balancing on a Collection

You can disable balancing for a specific collection with thesh.disableBalancing() method. You may want to disable thebalancer for a specific collection to support maintenance operations oratypical workloads, for example, during data ingestions or data exports.

When you disable balancing on a collection, MongoDB will not interrupt inprogress migrations.

To disable balancing on a collection, connect to a mongoswith the mongo shell and call thesh.disableBalancing() method.

For example:

sh.disableBalancing("students.grades")

The sh.disableBalancing() method accepts as its parameter thefull namespace of the collection.

Enable Balancing on a Collection

You can enable balancing for a specific collection with thesh.enableBalancing() method.

When you enable balancing for a collection, MongoDB will not _immediately_begin balancing data. However, if the data in your sharded collection isnot balanced, MongoDB will be able to begin distributing the data moreevenly.

To enable balancing on a collection, connect to a mongoswith the mongo shell and call thesh.enableBalancing() method.

For example:

sh.enableBalancing("students.grades")

The sh.enableBalancing() method accepts as its parameter thefull namespace of the collection.

Confirm Balancing is Enabled or Disabled

To confirm whether balancing for a collection is enabled or disabled,query the collections collection in the config database for thecollection namespace and check the noBalance field. Forexample:

db.getSiblingDB("config").collections.findOne({_id : "students.grades"}).noBalance;

This operation will return a null error, true, false, or no output:

A null error indicates the collection namespace is incorrect.
If the result is true, balancing is disabled.
If the result is false, balancing is enabled currently but has beendisabled in the past for the collection. Balancing of this collectionwill begin the next time the balancer runs.
If the operation returns no output, balancing is enabled currently andhas never been disabled in the past for this collection. Balancing ofthis collection will begin the next time the balancer runs.

New in version 3.0.0: You can also see if the balancer is enabled usingsh.status(). Thecurrently-enabled field indicates if thebalancer is enabled.

Change Replication Behavior for Chunk Migration

Secondary Throttle

During chunk migration, the _secondaryThrottle value determineswhen the migration proceeds with next document in the chunk.

In the config.settings collection:

If the _secondaryThrottle setting for the balancer is set to awrite concern, each document move during chunk migration must receivethe requested acknowledgement before proceeding with the nextdocument.
If the _secondaryThrottle setting for the balancer is set totrue, each document move during chunk migration must receiveacknowledgement from at least one secondary before the migrationproceeds with the next document in the chunk. This is equivalent to awrite concern of { w: 2 }.
If the _secondaryThrottle setting is unset, the migration processdoes not wait for replication to a secondary and instead continueswith the next document.

Default behavior for WiredTiger startingin MongoDB 3.4.

To change the _secondaryThrottle setting, connect to amongos instance and directly update the_secondaryThrottle value in the settings collectionof the config database. For example, from amongo shell connected to a mongos, issuethe following command:

use config
db.settings.update(
   { "_id" : "balancer" },
   { $set : { "_secondaryThrottle" : { "w": "majority" }  } },
   { upsert : true }
)

The effects of changing the _secondaryThrottle setting may not beimmediate. To ensure an immediate effect, stop and restart the balancerto enable the selected value of _secondaryThrottle.

For more information on the replication behavior during various stepsof chunk migration, see Chunk Migration and Replication.

For the moveChunk command, you can use the command’s_secondaryThrottle and writeConcern options to specify thebehavior during the command. For details, see moveChunkcommand.

Wait for Delete

The _waitForDelete setting of the balancer and themoveChunk command affects how the balancer migratesmultiple chunks from a shard. By default, the balancer does not waitfor the on-going migration’s delete phase to complete before startingthe next chunk migration. To have the delete phase block the startof the next chunk migration, you can set the _waitForDelete totrue.

For details on chunk migration, see Chunk Migration.For details on the chunk migration queuing behavior, seeAsynchronous Chunk Migration Cleanup.

The _waitForDelete is generally for internal testing purposes. Tochange the balancer’s _waitForDelete value:

Connect to a mongos instance.
Update the _waitForDelete value in the settingscollection of the config database. Forexample:

use config
db.settings.update(
   { "_id" : "balancer" },
   { $set : { "_waitForDelete" : true } },
   { upsert : true }
)

Once set to true, to revert to the default behavior:

Connect to a mongos instance.
Update or unset the _waitForDelete field in thesettings collection of the config database:

use config
db.settings.update(
   { "_id" : "balancer", "_waitForDelete": true },
   { $unset : { "_waitForDelete" : "" } }
)

Change the Maximum Storage Size for a Given Shard

By default shards have no constraints in storage size. However, you can set amaximum storage size for a given shard in the sharded cluster. Whenselecting potential destination shards, the balancer ignores shardswhere a migration would exceed the configured maximum storage size.

The shards collection in the configdatabase stores configuration data related to shards.

{ "_id" : "shard0000", "host" : "shard1.example.com:27100" }
{ "_id" : "shard0001", "host" : "shard2.example.com:27200" }

To limit the storage size for a given shard, use thedb.collection.updateOne() method with the $set operator tocreate the maxSize field and assign it an integer value. ThemaxSize field represents the maximum storage size for the shard inmegabytes.

The following operation sets a maximum size on a shard of 1024 megabytes:

config = db.getSiblingDB("config")
config.shards.updateOne( { "_id" : "<shard>"}, { $set : { "maxSize" : 1024 } } )

This value includes the mapped size of all data files on theshard, including the local and admin databases.

By default, maxSize is not specified, allowing shards to consume thetotal amount of available space on their machines if necessary.

You can also set maxSize when adding a shard.

To set maxSize when adding a shard, set the addShardcommand’s maxSize parameter to the maximum size in megabytes. Thefollowing command run in the mongo shell adds a shard with amaximum size of 125 megabytes:

config = db.getSiblingDB("config")
config.runCommand( { addshard : "example.net:34008", maxSize : 125 } )