Trying to do a bulk upsert with Mongoose. What's the cleanest way to do this?

Question

I have a collection that holds documents that contains three fields: first_name, last_name, and age. I\'m trying to figure out what query in Mongoose I can use to do a bulk

Answer 1

I tried @magnap's solution above and found that it was overwriting the currently existing documents that I simply wanted to update. Rather than updating the fields I set in updates.updateOne, it was selecting the document and replacing all of its fields with the ones specified in .update.

I eventually had to use $set in my update method to resolve this. Here's what my controller ended up looking like:

const { ObjectId } = require('mongodb');

exports.bulkUpsert = (req, res, next) => {
     const { updates } = req.body;
     const bulkOps = updates.map(update => ({
         updateOne: {
             filter: { _id: ObjectId(update.id) },
             // Where field is the field you want to update
             update: { $set: { field: update.field } },
             upsert: true
          }
      }));
    // where Model is the name of your model
    return Model.collection
        .bulkWrite(bulkOps)
        .then(results => res.json(results))
        .catch(err => next(err));
};

This works with Mongoose 5.1.2.

Answer 2

(mongoose@4.9.1, mongodb@3.4.2)

After struggling with Mongoose API poor documentation, I solved the bulk upsert tweaking updateOne:{} operation in the bulkWrite() method.

A couple of undocumented things to consider:

// suppose:
var GasStation = mongoose.model('gasstation', gasStationsSchema);
var bulkOps = [ ];

// for ( ... each gasStation to upsert ...) {
  let gasStation = { country:'a', localId:'b', xyz:'c' };
  // [populate gasStation as needed]
  // Each document should look like this: (note the 'upsert': true)
  let upsertDoc = {
    'updateOne': {
      'filter': { 'country': gasStation.country, 'localId': gasStation.localId },
      'update': gasStation,
      'upsert': true
    }};
  bulkOps.push(upsertDoc);
// end for loop

// now bulkWrite (note the use of 'Model.collection')
GasStation.collection.bulkWrite(bulkOps)
  .then( bulkWriteOpResult => {
    console.log('BULK update OK');
    console.log(JSON.stringify(bulkWriteOpResult, null, 2));
  })
  .catch( err => {
    console.log('BULK update error');
    console.log(JSON.stringify(err, null, 2));
  });

The two key things here are incomplete API documentation issues (at the time of writing, at least):

• 'upsert': true in each document. This is not documented in Mongoose API (), which often refers to node-mongodb-native driver. Looking at updateOne in this driver, you could think to add 'options':{'upsert': true}, but, no... that won't do. I also tried to add both cases to the bulkWrite(,[options],) argument, with no effect either.
• GasStation.collection.bulkWrite(). Although Mongoose bulkWrite() method claims it should be called Model.bulkWrite() (in this case, GasStation.bulkWrite()), that will trigger MongoError: Unknown modifier: $__. So, Model.collection.bulkWrite() must be used.

Additionally, note:

• You don't need to use the $set mongo operator in the updateOne.update field, since mongoose handles it in case of upsert (see bulkWrite() comments in example).
• Note that my unique index in the schema (needed for upsert to work properly) is defined as:

gasStationsSchema.index({ country: 1, localId: 1 }, { unique: true });

Hope it helps.

==> EDIT: (Mongoose 5?)

As noticed by @JustinSmith, the $set operator added by Mongoose doesn't seem to be working anymore. Maybe it's because of Mongoose 5?

In any case, using $set explicitly should do:

'update': { '$set': gasStation },

Answer 3

check this i hope this will helpfull for you link

link2

I think you are looking for the

Bulk.find().upsert().update()

yo can use this

bulk = db.yourCollection.initializeUnorderedBulkOp();
for (<your for statement>) {
    bulk.find({ID: <your id>, HASH: <your hash>}).upsert().update({<your update fields>});
}
bulk.execute(<your callback>)

• If it finds one, it will update that document using {}
• Otherwise, it will create a new document

Answer 4

I have released a small plugin for Mongoose that exposes a static upsertMany method to perform bulk upsert operations with a promise interface. This should provide a very clean way of doing bulk upserts with Mongoose, while retaining schema validation etc:

MyModel.upsertMany(items, ['matchField', 'other.nestedMatchField']);

You can find this plugin on npm or Github:

https://github.com/meanie/mongoose-upsert-many https://www.npmjs.com/package/@meanie/mongoose-upsert-many

Answer 5

Found the official solution on: https://docs.mongodb.com/manual/reference/method/Bulk.find.upsert/

And Mongoose also supports same chain.

Bulk.find(<query>).upsert().update(<update>);
Bulk.find(<query>).upsert().updateOne(<update>);
Bulk.find(<query>).upsert().replaceOne(<replacement>);

Tested it works:

BulkWriteResult {
  result:
   { ok: 1,
     writeErrors: [],
     writeConcernErrors: [],
     insertedIds: [],
     nInserted: 0,
     nUpserted: 1,
     nMatched: 4186,
     nModified: 0,
     nRemoved: 0,
     upserted: [ [Object] ] } }

Answer 6

Thank @maganap. I used his/her answer and reached below concise approach:

await Model.bulkWrite(docs.map(doc => ({
    updateOne: {
        filter: {id: doc.id},
        update: doc,
        upsert: true
    }
})))

---

Or more verbose:

const bulkOps = docs.map(doc => ({
    updateOne: {
        filter: {id: doc.id},
        update: doc,
        upsert: true
    }
}))

Model.bulkWrite(bulkOps)
        .then(bulkWriteOpResult => console.log('BULK update OK:', bulkWriteOpResult))
        .catch(console.error.bind(console, 'BULK update error:')