Bulk Write Operations

This tutorial explains how to take advantage of PyMongo’s bulkwrite operation features. Executing write operations in batchesreduces the number of network round trips, increasing writethroughput.

Bulk Insert

New in version 2.6.

A batch of documents can be inserted by passing a list to theinsert_many() method. PyMongowill automatically split the batch into smaller sub-batches based onthe maximum message size accepted by MongoDB, supporting very largebulk insert operations.

>>> import pymongo
>>> db = pymongo.MongoClient().bulk_example
>>> db.test.insert_many([{'i': i} for i in range(10000)]).inserted_ids
[...]
>>> db.test.count_documents({})
10000

Mixed Bulk Write Operations

New in version 2.7.

PyMongo also supports executing mixed bulk write operations. A batchof insert, update, and remove operations can be executed together usingthe bulk write operations API.

Ordered Bulk Write Operations

Ordered bulk write operations are batched and sent to the server in theorder provided for serial execution. The return value is an instance ofBulkWriteResult describing the type and countof operations performed.

>>> from pprint import pprint
>>> from pymongo import InsertOne, DeleteMany, ReplaceOne, UpdateOne
>>> result = db.test.bulk_write([
...     DeleteMany({}),  # Remove all documents from the previous example.
...     InsertOne({'_id': 1}),
...     InsertOne({'_id': 2}),
...     InsertOne({'_id': 3}),
...     UpdateOne({'_id': 1}, {'$set': {'foo': 'bar'}}),
...     UpdateOne({'_id': 4}, {'$inc': {'j': 1}}, upsert=True),
...     ReplaceOne({'j': 1}, {'j': 2})])
>>> pprint(result.bulk_api_result)
{'nInserted': 3,
 'nMatched': 2,
 'nModified': 2,
 'nRemoved': 10000,
 'nUpserted': 1,
 'upserted': [{u'_id': 4, u'index': 5}],
 'writeConcernErrors': [],
 'writeErrors': []}

Warning

nModified is only reported by MongoDB 2.6 and later. Whenconnected to an earlier server version, or in certain mixed version shardingconfigurations, PyMongo omits this field from the results of a bulkwrite operation.

The first write failure that occurs (e.g. duplicate key error) aborts theremaining operations, and PyMongo raisesBulkWriteError. The details attibute ofthe exception instance provides the execution results up until the failureoccurred and details about the failure - including the operation that causedthe failure.

>>> from pymongo import InsertOne, DeleteOne, ReplaceOne
>>> from pymongo.errors import BulkWriteError
>>> requests = [
...     ReplaceOne({'j': 2}, {'i': 5}),
...     InsertOne({'_id': 4}),  # Violates the unique key constraint on _id.
...     DeleteOne({'i': 5})]
>>> try:
...     db.test.bulk_write(requests)
... except BulkWriteError as bwe:
...     pprint(bwe.details)
...
{'nInserted': 0,
 'nMatched': 1,
 'nModified': 1,
 'nRemoved': 0,
 'nUpserted': 0,
 'upserted': [],
 'writeConcernErrors': [],
 'writeErrors': [{u'code': 11000,
                  u'errmsg': u'...E11000...duplicate key error...',
                  u'index': 1,...
                  u'op': {'_id': 4}}]}

Unordered Bulk Write Operations

Unordered bulk write operations are batched and sent to the server inarbitrary order where they may be executed in parallel. Any errorsthat occur are reported after all operations are attempted.

In the next example the first and third operations fail due to the uniqueconstraint on _id. Since we are doing unordered execution the secondand fourth operations succeed.

>>> requests = [
...     InsertOne({'_id': 1}),
...     DeleteOne({'_id': 2}),
...     InsertOne({'_id': 3}),
...     ReplaceOne({'_id': 4}, {'i': 1})]
>>> try:
...     db.test.bulk_write(requests, ordered=False)
... except BulkWriteError as bwe:
...     pprint(bwe.details)
...
{'nInserted': 0,
 'nMatched': 1,
 'nModified': 1,
 'nRemoved': 1,
 'nUpserted': 0,
 'upserted': [],
 'writeConcernErrors': [],
 'writeErrors': [{u'code': 11000,
                  u'errmsg': u'...E11000...duplicate key error...',
                  u'index': 0,...
                  u'op': {'_id': 1}},
                 {u'code': 11000,
                  u'errmsg': u'...E11000...duplicate key error...',
                  u'index': 2,...
                  u'op': {'_id': 3}}]}

Write Concern

Bulk operations are executed with thewrite_concern of the collection theyare executed against. Write concern errors (e.g. wtimeout) will be reportedafter all operations are attempted, regardless of execution order.

• ::

>>> from pymongo import WriteConcern
>>> coll = db.get_collection(
...     'test', write_concern=WriteConcern(w=3, wtimeout=1))
>>> try:
...     coll.bulk_write([InsertOne({'a': i}) for i in range(4)])
... except BulkWriteError as bwe:
...     pprint(bwe.details)
...
{'nInserted': 4,
 'nMatched': 0,
 'nModified': 0,
 'nRemoved': 0,
 'nUpserted': 0,
 'upserted': [],
 'writeConcernErrors': [{u'code': 64...
                         u'errInfo': {u'wtimeout': True},
                         u'errmsg': u'waiting for replication timed out'}],
 'writeErrors': []}