Correctly inserting and/or updating many datasets to MongoDB (using mongoose)?

问题

So from time to time I get new exports of a cities database of POIs and info about them and I want to have all that data in my MongoDB with a Loopback-API on it. Therefore I reduce the data to my desired structure and try to import it.

For the first time I receive such an export, I can simply insert the data with insertMany().

When I get a new export, it means that it includes updated POIs which I actually want my existing POIs to be replaced with that new data. So I thought I'd use updateMany() but I could'nt figure out how I'd do that in my case.

Here's what I have so far:

const fs = require('fs');
const mongoose = require('mongoose');

const data = JSON.parse(fs.readFileSync('data.json', 'utf8'));

// Connect to database
mongoose.connect('mongodb://localhost/test', {
  useMongoClient: true
}, (err) => {
  if (err) console.log('Error', err);
});

// Define schema
let poiSchema = new mongoose.Schema({
  _id:          Number,
  name:         String,
  geo:          String,
  street:       String,
  housenumber:  String,
  phone:        String,
  website:      String,
  email:        String,
  category:     String
});

// Set model
let poi = mongoose.model('poi', poiSchema);

// Generate specified data from given export
let reducedData = data['ogr:FeatureCollection']['gml:featureMember'].reduce((endData, iteratedItem) => {
  endData = endData.length > 0 ? endData : [];

  endData.push({
    _id:          iteratedItem['service']['fieldX'],
    name:         iteratedItem['service']['fieldX'],
    geo:          iteratedItem['service']['fieldX']['fieldY']['fieldZ'],
    street:       iteratedItem['service']['fieldX'],
    housenumber:  iteratedItem['service']['fieldX'],
    phone:        iteratedItem['service']['fieldX'],
    website:      iteratedItem['service']['fieldX'],
    email:        iteratedItem['service']['fieldX'],
    category:     iteratedItem['service']['fieldX']
  });

  return endData;
}, []);

//
// HERE: ?!?!? Insert/update reduced data in MongoDB collection ?!?!?
//

mongoose.disconnect();

So I just want to update everything that has changed.

Of course if I leave it to insertMany() it fails due to dup key.

回答1:

For the second time, use mongo's update command with upsert set to true.

db.collection.update(query, update, options)

In the query pass the _id ,in update pass the object and in option set upsert to true. This will update the document if it exists creates a new document if that doesn't exist.

来源：https://stackoverflow.com/questions/47697822/correctly-inserting-and-or-updating-many-datasets-to-mongodb-using-mongoose

标签

mongodb

import

insert-update