MongoDB: Combine data from multiple collections into one..how?
How can I (in MongoDB) combine data from multiple collections into one collection?
Can I use map-reduce and if so then how?
I would greatly appreciate some
-
Starting
Mongo 4.4, we can achieve this join within an aggregation pipeline by coupling the new $unionWith aggregation stage with$group's new $accumulator operator:// > db.users.find() // [{ user: 1, name: "x" }, { user: 2, name: "y" }] // > db.books.find() // [{ user: 1, book: "a" }, { user: 1, book: "b" }, { user: 2, book: "c" }] // > db.movies.find() // [{ user: 1, movie: "g" }, { user: 2, movie: "h" }, { user: 2, movie: "i" }] db.users.aggregate([ { $unionWith: "books" }, { $unionWith: "movies" }, { $group: { _id: "$user", user: { $accumulator: { accumulateArgs: ["$name", "$book", "$movie"], init: function() { return { books: [], movies: [] } }, accumulate: function(user, name, book, movie) { if (name) user.name = name; if (book) user.books.push(book); if (movie) user.movies.push(movie); return user; }, merge: function(userV1, userV2) { if (userV2.name) userV1.name = userV2.name; userV1.books.concat(userV2.books); userV1.movies.concat(userV2.movies); return userV1; }, lang: "js" } } }} ]) // { _id: 1, user: { books: ["a", "b"], movies: ["g"], name: "x" } } // { _id: 2, user: { books: ["c"], movies: ["h", "i"], name: "y" } }$unionWith combines records from the given collection within documents already in the aggregation pipeline. After the 2 union stages, we thus have all users, books and movies records within the pipeline.
We then
$grouprecords by$userand accumulate items using the $accumulator operator allowing custom accumulations of documents as they get grouped:- the fields we're interested in accumulating are defined with
accumulateArgs. initdefines the state that will be accumulated as we group elements.- the
accumulatefunction allows performing a custom action with a record being grouped in order to build the accumulated state. For instance, if the item being grouped has thebookfield defined, then we update thebookspart of the state. mergeis used to merge two internal states. It's only used for aggregations running on sharded clusters or when the operation exceeds memory limits.
- the fields we're interested in accumulating are defined with
讨论(0) -
Mongorestore has this feature of appending on top of whatever is already in the database, so this behavior could be used for combining two collections:
- mongodump collection1
- collection2.rename(collection1)
- mongorestore
Didn't try it yet, but it might perform faster than the map/reduce approach.
讨论(0) -
You've to do that in your application layer. If you're using an ORM, it could use annotations (or something similar) to pull references that exist in other collections. I only have worked with Morphia, and the @Reference annotation fetches the referenced entity when queried, so I am able to avoid doing it myself in the code.
讨论(0) -
MongoDB 3.2 now allows one to combine data from multiple collections into one through the $lookup aggregation stage. As a practical example, lets say that you have data about books split into two different collections.
First collection, called
books, having the following data:{ "isbn": "978-3-16-148410-0", "title": "Some cool book", "author": "John Doe" } { "isbn": "978-3-16-148999-9", "title": "Another awesome book", "author": "Jane Roe" }And the second collection, called
books_selling_data, having the following data:{ "_id": ObjectId("56e31bcf76cdf52e541d9d26"), "isbn": "978-3-16-148410-0", "copies_sold": 12500 } { "_id": ObjectId("56e31ce076cdf52e541d9d28"), "isbn": "978-3-16-148999-9", "copies_sold": 720050 } { "_id": ObjectId("56e31ce076cdf52e541d9d29"), "isbn": "978-3-16-148999-9", "copies_sold": 1000 }To merge both collections is just a matter of using $lookup in the following way:
db.books.aggregate([{ $lookup: { from: "books_selling_data", localField: "isbn", foreignField: "isbn", as: "copies_sold" } }])After this aggregation, the
bookscollection will look like the following:{ "isbn": "978-3-16-148410-0", "title": "Some cool book", "author": "John Doe", "copies_sold": [ { "_id": ObjectId("56e31bcf76cdf52e541d9d26"), "isbn": "978-3-16-148410-0", "copies_sold": 12500 } ] } { "isbn": "978-3-16-148999-9", "title": "Another awesome book", "author": "Jane Roe", "copies_sold": [ { "_id": ObjectId("56e31ce076cdf52e541d9d28"), "isbn": "978-3-16-148999-9", "copies_sold": 720050 }, { "_id": ObjectId("56e31ce076cdf52e541d9d28"), "isbn": "978-3-16-148999-9", "copies_sold": 1000 } ] }It is important to note a few things:
- The "from" collection, in this case
books_selling_data, cannot be sharded. - The "as" field will be an array, as the example above.
- Both "localField" and "foreignField" options on the $lookup stage will be treated as null for matching purposes if they don't exist in their respective collections (the $lookup docs has a perfect example about that).
So, as a conclusion, if you want to consolidate both collections, having, in this case, a flat copies_sold field with the total copies sold, you will have to work a little bit more, probably using an intermediary collection that will, then, be $out to the final collection.
讨论(0) - The "from" collection, in this case
-
use multiple $lookup for multiple collections in aggregation
query:
db.getCollection('servicelocations').aggregate([ { $match: { serviceLocationId: { $in: ["36728"] } } }, { $lookup: { from: "orders", localField: "serviceLocationId", foreignField: "serviceLocationId", as: "orders" } }, { $lookup: { from: "timewindowtypes", localField: "timeWindow.timeWindowTypeId", foreignField: "timeWindowTypeId", as: "timeWindow" } }, { $lookup: { from: "servicetimetypes", localField: "serviceTimeTypeId", foreignField: "serviceTimeTypeId", as: "serviceTime" } }, { $unwind: "$orders" }, { $unwind: "$serviceTime" }, { $limit: 14 } ])result:
{ "_id" : ObjectId("59c3ac4bb7799c90ebb3279b"), "serviceLocationId" : "36728", "regionId" : 1.0, "zoneId" : "DXBZONE1", "description" : "AL HALLAB REST EMIRATES MALL", "locationPriority" : 1.0, "accountTypeId" : 1.0, "locationType" : "SERVICELOCATION", "location" : { "makani" : "", "lat" : 25.119035, "lng" : 55.198694 }, "deliveryDays" : "MTWRFSU", "timeWindow" : [ { "_id" : ObjectId("59c3b0a3b7799c90ebb32cde"), "timeWindowTypeId" : "1", "Description" : "MORNING", "timeWindow" : { "openTime" : "06:00", "closeTime" : "08:00" }, "accountId" : 1.0 }, { "_id" : ObjectId("59c3b0a3b7799c90ebb32cdf"), "timeWindowTypeId" : "1", "Description" : "MORNING", "timeWindow" : { "openTime" : "09:00", "closeTime" : "10:00" }, "accountId" : 1.0 }, { "_id" : ObjectId("59c3b0a3b7799c90ebb32ce0"), "timeWindowTypeId" : "1", "Description" : "MORNING", "timeWindow" : { "openTime" : "10:30", "closeTime" : "11:30" }, "accountId" : 1.0 } ], "address1" : "", "address2" : "", "phone" : "", "city" : "", "county" : "", "state" : "", "country" : "", "zipcode" : "", "imageUrl" : "", "contact" : { "name" : "", "email" : "" }, "status" : "ACTIVE", "createdBy" : "", "updatedBy" : "", "updateDate" : "", "accountId" : 1.0, "serviceTimeTypeId" : "1", "orders" : [ { "_id" : ObjectId("59c3b291f251c77f15790f92"), "orderId" : "AQ18O1704264", "serviceLocationId" : "36728", "orderNo" : "AQ18O1704264", "orderDate" : "18-Sep-17", "description" : "AQ18O1704264", "serviceType" : "Delivery", "orderSource" : "Import", "takenBy" : "KARIM", "plannedDeliveryDate" : ISODate("2017-08-26T00:00:00.000Z"), "plannedDeliveryTime" : "", "actualDeliveryDate" : "", "actualDeliveryTime" : "", "deliveredBy" : "", "size1" : 296.0, "size2" : 3573.355, "size3" : 240.811, "jobPriority" : 1.0, "cancelReason" : "", "cancelDate" : "", "cancelBy" : "", "reasonCode" : "", "reasonText" : "", "status" : "", "lineItems" : [ { "ItemId" : "BNWB020", "size1" : 15.0, "size2" : 78.6, "size3" : 6.0 }, { "ItemId" : "BNWB021", "size1" : 20.0, "size2" : 252.0, "size3" : 11.538 }, { "ItemId" : "BNWB023", "size1" : 15.0, "size2" : 285.0, "size3" : 16.071 }, { "ItemId" : "CPMW112", "size1" : 3.0, "size2" : 25.38, "size3" : 1.731 }, { "ItemId" : "MMGW001", "size1" : 25.0, "size2" : 464.375, "size3" : 46.875 }, { "ItemId" : "MMNB218", "size1" : 50.0, "size2" : 920.0, "size3" : 60.0 }, { "ItemId" : "MMNB219", "size1" : 50.0, "size2" : 630.0, "size3" : 40.0 }, { "ItemId" : "MMNB220", "size1" : 50.0, "size2" : 416.0, "size3" : 28.846 }, { "ItemId" : "MMNB270", "size1" : 50.0, "size2" : 262.0, "size3" : 20.0 }, { "ItemId" : "MMNB302", "size1" : 15.0, "size2" : 195.0, "size3" : 6.0 }, { "ItemId" : "MMNB373", "size1" : 3.0, "size2" : 45.0, "size3" : 3.75 } ], "accountId" : 1.0 }, { "_id" : ObjectId("59c3b291f251c77f15790f9d"), "orderId" : "AQ137O1701240", "serviceLocationId" : "36728", "orderNo" : "AQ137O1701240", "orderDate" : "18-Sep-17", "description" : "AQ137O1701240", "serviceType" : "Delivery", "orderSource" : "Import", "takenBy" : "KARIM", "plannedDeliveryDate" : ISODate("2017-08-26T00:00:00.000Z"), "plannedDeliveryTime" : "", "actualDeliveryDate" : "", "actualDeliveryTime" : "", "deliveredBy" : "", "size1" : 28.0, "size2" : 520.11, "size3" : 52.5, "jobPriority" : 1.0, "cancelReason" : "", "cancelDate" : "", "cancelBy" : "", "reasonCode" : "", "reasonText" : "", "status" : "", "lineItems" : [ { "ItemId" : "MMGW001", "size1" : 25.0, "size2" : 464.38, "size3" : 46.875 }, { "ItemId" : "MMGW001-F1", "size1" : 3.0, "size2" : 55.73, "size3" : 5.625 } ], "accountId" : 1.0 }, { "_id" : ObjectId("59c3b291f251c77f15790fd8"), "orderId" : "AQ110O1705036", "serviceLocationId" : "36728", "orderNo" : "AQ110O1705036", "orderDate" : "18-Sep-17", "description" : "AQ110O1705036", "serviceType" : "Delivery", "orderSource" : "Import", "takenBy" : "KARIM", "plannedDeliveryDate" : ISODate("2017-08-26T00:00:00.000Z"), "plannedDeliveryTime" : "", "actualDeliveryDate" : "", "actualDeliveryTime" : "", "deliveredBy" : "", "size1" : 60.0, "size2" : 1046.0, "size3" : 68.0, "jobPriority" : 1.0, "cancelReason" : "", "cancelDate" : "", "cancelBy" : "", "reasonCode" : "", "reasonText" : "", "status" : "", "lineItems" : [ { "ItemId" : "MMNB218", "size1" : 50.0, "size2" : 920.0, "size3" : 60.0 }, { "ItemId" : "MMNB219", "size1" : 10.0, "size2" : 126.0, "size3" : 8.0 } ], "accountId" : 1.0 } ], "serviceTime" : { "_id" : ObjectId("59c3b07cb7799c90ebb32cdc"), "serviceTimeTypeId" : "1", "serviceTimeType" : "nohelper", "description" : "", "fixedTime" : 30.0, "variableTime" : 0.0, "accountId" : 1.0 } }讨论(0)
加载中...
- 热议问题