Following an example from the mongodb manual for nodejs, I am finding all documents from a db as follows
mongo.Db.connect(mongoUri, function (err, db) {
modern way
const all_data = [];
const cursor = Collection.find({})
await cursor.forEach(function(doc){
all_data.push(doc)
})
all_data // array with all the documents in the collection.
You can stream the results of a node.js native driver's query by calling stream() on the returned cursor:
var stream = collection.find().stream();
stream.on('data', function(doc) {
console.log(doc);
});
stream.on('error', function(err) {
console.log(err);
});
stream.on('end', function() {
console.log('All done!');
});
Would limiting the query be an option? Literally with db.collection.find().limit()? The limit is parsed before sending the command to the server, so it only scans the amount of data in your limit.
The easiest way is to use a Cursor
(reference):
var cursor = db.collection('test').find();
// Execute the each command, triggers for each document
cursor.each(function(err, item) {
// If the item is null then the cursor is exhausted/empty and closed
if(item == null) {
db.close(); // you may not want to close the DB if you have more code....
return;
}
// otherwise, do something with the item
});
If there's a lot of computation you need to do, you might consider whether a Map-Reduce (reference) would fit your needs as the code would execute on the DB server, rather than locally.