Avoid Aggregate 16MB Limit

后端 未结 2 1394
耶瑟儿~
耶瑟儿~ 2020-12-11 07:57

I have a collection of about 1M documents. Each document has internalNumber property and I need to get all internalNumbers in my node.js code.

相关标签:
2条回答
  • 2020-12-11 08:19

    For Casbah users:

    val pipeline = ...
    collection.aggregate(pipeline, AggregationOptions(batchSize = 500, outputMode = AggregationOptions.CURSOR)
    
    0 讨论(0)
  • 2020-12-11 08:27

    The problem is that the native driver differs from how the shell method is working by default in that the "shell" is actually returning a "cursor" object where the native driver needs this option "explicitly".

    Without a "cursor", .aggregate() returns a single BSON document as an array of documents, so we turn it into a cursor to avoid the limitation:

    let cursor = collection.aggregate(
      [{ "$group": { "_id": "$internalNumber" } }],
      { "cursor": { "batchSize": 500 } }
    );
    
    cursor.toArray((err,docs) => {
       // work with resuls
    });
    

    Then you can use regular methods like .toArray() to make the results a JavaScript array which on the 'client' does not share the same limitations, or other methods for iterating a "cursor".

    0 讨论(0)
提交回复
热议问题