Return limited number of records of a certain type, but unlimited number of other records?

Question

I have a query where I need to return 10 of \"Type A\" records, while returning all other records. How can I accomplish this?

Update: Admittedly, I

Answer 1

This is a classic case for subquery/join which is not supported by MongoDB. All joins and subquery-like operations need to be implemented in the application logic. So multiple queries is your best bet. Performance of the multiple query approach should be good if you have an index on type.

Alternatively you can write a single aggregation query minus the type-matching and limit clauses and then process the stream in your application logic to limit documents per type. This approach will be low on performance for large result sets because documents may be returned in random order. Your limiting logic will then need to traverse to the entire result set.

Answer 2

i guess you can use cursor.limit() on a cursor to specify the maximum number of documents the cursor will return. limit() is analogous to the LIMIT statement in a SQL database. You must apply limit() to the cursor before retrieving any documents from the database.

The limit function in the cursors can be used for limiting the number of records in the find.

I guess this example should help:

var myCursor = db.bios.find( );

db.bios.find().limit( 5 )

Answer 3

I don't think this is presently (2.6) possible to do with one aggregation pipeline. It's difficult to give a precise argument as to why not, but basically the aggregation pipeline performs transformations of streams of documents, one document at a time. There's no awareness within the pipeline of the state of the stream itself, which is what you'd need to determine that you've hit the limit for A's, B's, etc and need to drop further documents of the same type. $group does bring multiple documents together and allows their field values in aggregate to affect the resulting group document ($sum, $avg, etc.). Maybe this makes some sense, but it's necessarily not rigorous because there are simple operations you could add to make it possible to limit based on the types, e.g., adding a $push x accumulator to $group that only pushes the value if the array being pushed to has fewer than x elements.

Even if I did have a way to do it, I'd recommend just doing two aggregations. Keep it simple.

Answer 4

Problem

---

The results here are not impossible but are also possibly impractical. The general notes have been made that you cannot "slice" an array or otherwise "limit" the amount of results pushed onto one. And the method for doing this per "type" is essentially to use arrays.

The "impractical" part is usually about the number of results, where too large a result set is going to blow up the BSON document limit when "grouping". But, I'm going to consider this with some other recommendations on your "geo search" along with the ultimate goal to return 10 results of each "type" at most.

Principle

---

To first consider and understand the problem, let's look at a simplified "set" of data and the pipeline code necessary to return the "top 2 results" from each type:

{ "title": "Title 1", "author": "Author 1", "type": "Fiction", "distance": 1 },
{ "title": "Title 2", "author": "Author 2", "type": "Fiction", "distance": 2 },
{ "title": "Title 3", "author": "Author 3", "type": "Fiction", "distance": 3 },
{ "title": "Title 4", "author": "Author 4", "type": "Science", "distance": 1 },
{ "title": "Title 5", "author": "Author 5", "type": "Science", "distance": 2 },
{ "title": "Title 6", "author": "Author 6", "type": "Science", "distance": 3 },
{ "title": "Title 7", "author": "Author 7", "type": "Horror", "distance": 1 }

That's a simplified view of the data and somewhat representative of the state of documents after an initial query. Now comes the trick of how to use the aggregation pipeline to get the "nearest" two results for each "type":

db.books.aggregate([
    { "$sort": { "type": 1, "distance": 1 } },
    { "$group": {
        "_id": "$type",
        "1": { 
            "$first": {
                "_id": "$_id",
                "title": "$title",
                "author": "$author",
                "distance": "$distance"
            }
         },
         "books": {
             "$push": {
                "_id": "$_id",
                "title": "$title",
                "author": "$author",
                "distance": "$distance"
              }
         }
    }},
    { "$project": {
        "1": 1,
        "books": {
            "$cond": [
                { "$eq": [ { "$size": "$books" }, 1 ] },
                { "$literal": [false] },
                "$books"
            ]
        }
    }},
    { "$unwind": "$books" },
    { "$project": {
        "1": 1,
        "books": 1,
        "seen": { "$eq": [ "$1", "$books" ] }
    }},
    { "$sort": { "_id": 1, "seen": 1 } },
    { "$group": {
        "_id": "$_id",
        "1": { "$first": "$1" },
        "2": { "$first": "$books" },
        "books": {
            "$push": {
                "$cond": [ { "$not": "$seen" }, "$books", false ]
            }
        }
    }},
    { "$project": {
        "1": 1,
        "2": 2,
        "pos": { "$literal": [1,2] }
    }},
    { "$unwind": "$pos" },
    { "$group": {
        "_id": "$_id",
        "books": {
            "$push": {
                "$cond": [
                    { "$eq": [ "$pos", 1 ] },
                    "$1",
                    { "$cond": [
                        { "$eq": [ "$pos", 2 ] },
                        "$2",
                        false
                    ]}
                ]
            }
        }
    }},
    { "$unwind": "$books" },
    { "$match": { "books": { "$ne": false } } },
    { "$project": {
        "_id": "$books._id",
        "title": "$books.title",
        "author": "$books.author",
        "type": "$_id",
        "distance": "$books.distance",
        "sortOrder": {
            "$add": [
                { "$cond": [ { "$eq": [ "$_id", "Fiction" ] }, 1, 0 ] },
                { "$cond": [ { "$eq": [ "$_id", "Science" ] }, 0, 0 ] },
                { "$cond": [ { "$eq": [ "$_id", "Horror" ] }, 3, 0 ] }
            ]
        }
    }},
    { "$sort": { "sortOrder": 1 } }
])

Of course that is just two results, but it outlines the process for getting n results, which naturally is done in generated pipeline code. Before moving onto the code the process deserves a walk through.

After any query, the first thing to do here is $sort the results, and this you want to basically do by both the "grouping key" which is the "type" and by the "distance" so that the "nearest" items are on top.

The reason for this is shown in the $group stages that will repeat. What is done is essentially "popping the $first result off of each grouping stack. So other documents are not lost, they are placed in an array using $push.

Just to be safe, the next stage is really only required after the "first step", but could optionally be added for similar filtering in the repetition. The main check here is that the resulting "array" is larger than just one item. Where it is not, the contents are replaced with a single value of false. The reason for which is about to become evident.

After this "first step" the real repetition cycle beings, where that array is then "de-normalized" with $unwind and then a $project made in order to "match" the document that has been last "seen".

As only one of the documents will match this condition the results are again "sorted" in order to float the "unseen" documents to the top, while of course maintaining the grouping order. The next thing is similar to the first $group step, but where any kept positions are maintained and the "first unseen" document is "popped off the stack" again.

The document that was "seen" is then pushed back to the array not as itself but as a value of false. This is not going to match the kept value and this is generally the way to handle this without being "destructive" to the array contents where you don't want the operations to fail should there not be enough matches to cover the n results required.

Cleaning up when complete, the next "projection" adds an array to the final documents now grouped by "type" representing each position in the n results required. When this array is unwound, the documents can again be grouped back together, but now all in a single array that possibly contains several false values but is n elements long.

Finally unwind the array again, use $match to filter out the false values, and project to the required document form.

Practicality

---

The problem as stated earlier is with the number of results being filtered as there is a real limit on the number of results that can be pushed into an array. That is mostly the BSON limit, but you also don't really want 1000's of items even if that is still under the limit.

The trick here is keeping the initial "match" small enough that the "slicing operations" becomes practical. There are some things with the $geoNear pipeline process that can make this a possibility.

The obvious is limit. By default this is 100 but you clearly want to have something in the range of:

(the number of categories you can possibly match) X ( required matches )

But if this is essentially a number not in the 1000's then there is already some help here.

The others are maxDistance and minDistance, where essentially you put upper and lower bounds on how "far out" to search. The max bound is the general limiter while the min bound is useful when "paging", which is the next helper.

When "upwardly paging", you can use the query argument in order to exclude the _id values of documents "already seen" using the $nin query. In much the same way, the minDistance can be populated with the "last seen" largest distance, or at least the smallest largest distance by "type". This allows some concept of filtering out things that have already been "seen" and getting another page.

Really a topic in itself, but those are the general things to look for in reducing that initial match in order to make the process practical.

Implementing

---

The general problem of returning "10 results at most, per type" is clearly going to want some code in order to generate the pipeline stages. No-one wants to type that out, and practically speaking you will probably want to change that number at some point.

So now to the code that can generate the monster pipeline. All code in JavaScript, but easy to translate in principles:

var coords = [-118.09771, 33.89244];

var key = "$type";
var val = {
    "_id": "$_id",
    "title": "$title",
    "author": "$author",
    "distance": "$distance"
};
var maxLen = 10;

var stack = [];
var pipe = [];
var fproj = { "$project": { "pos": { "$literal": []  } } };

pipe.push({ "$geoNear": {
    "near": coords, 
    "distanceField": "distance", 
    "spherical": true
}});

pipe.push({ "$sort": {
    "type": 1, "distance": 1
}});

for ( var x = 1; x <= maxLen; x++ ) {

    fproj["$project"][""+x] = 1;
    fproj["$project"]["pos"]["$literal"].push( x );

    var rec = {
        "$cond": [ { "$eq": [ "$pos", x ] }, "$"+x ]
    };
    if ( stack.length == 0 ) {
        rec["$cond"].push( false );
    } else {
        lval = stack.pop();
        rec["$cond"].push( lval );
    }

    stack.push( rec );

    if ( x == 1) {
        pipe.push({ "$group": {
           "_id": key,
           "1": { "$first": val },
           "books": { "$push": val }
        }});
        pipe.push({ "$project": {
           "1": 1,
           "books": {
               "$cond": [
                    { "$eq": [ { "$size": "$books" }, 1 ] },
                    { "$literal": [false] },
                    "$books"
               ]
           }
        }});
    } else {
        pipe.push({ "$unwind": "$books" });
        var proj = {
            "$project": {
                "books": 1
            }
        };

        proj["$project"]["seen"] = { "$eq": [ "$"+(x-1), "$books" ] };

        var grp = {
            "$group": {
                "_id": "$_id",
                "books": {
                    "$push": {
                        "$cond": [ { "$not": "$seen" }, "$books", false ]
                    }
                }
            }
        };

        for ( n=x; n >= 1; n-- ) {
            if ( n != x ) 
                proj["$project"][""+n] = 1;
            grp["$group"][""+n] = ( n == x ) ? { "$first": "$books" } : { "$first": "$"+n };
        }

        pipe.push( proj );
        pipe.push({ "$sort": { "_id": 1, "seen": 1 } });
        pipe.push(grp);
    }
}

pipe.push(fproj);
pipe.push({ "$unwind": "$pos" });
pipe.push({
    "$group": {
        "_id": "$_id",
        "msgs": { "$push": stack[0] }
    }
});
pipe.push({ "$unwind": "$books" });
pipe.push({ "$match": { "books": { "$ne": false } }});
pipe.push({
    "$project": {
        "_id": "$books._id",
        "title": "$books.title",
        "author": "$books.author",
        "type": "$_id",
        "distance": "$books",
        "sortOrder": {
            "$add": [
                { "$cond": [ { "$eq": [ "$_id", "Fiction" ] }, 1, 0 ] },
                { "$cond": [ { "$eq": [ "$_id", "Science" ] }, 0, 0 ] },
                { "$cond": [ { "$eq": [ "$_id", "Horror" ] }, 3, 0 ] },
            ]
        }
    }
});
pipe.push({ "$sort": { "sortOrder": 1, "distance": 1 } });

Alternate

---

Of course the end result here and the general problem with all above is that you really only want the "top 10" of each "type" to return. The aggregation pipeline will do it, but at the cost of keeping more than 10 and then "popping off the stack" until 10 is reached.

An alternate approach is to "brute force" this with mapReduce and "globally scoped" variables. Not as nice since the results all in arrays, but it may be a practical approach:

db.collection.mapReduce(
    function () {

        if ( !stash.hasOwnProperty(this.type) ) {
            stash[this.type] = [];
        }

        if ( stash[this.type.length < maxLen ) {
            stash[this.type].push({
                "title": this.title,
                "author": this.author,
                "type": this.type,
                "distance": this.distance
            });
            emit( this.type, 1 );
        }

    },
    function(key,values) {
        return 1;   // really just want to keep the keys
    },
    { 
        "query": {
            "location": {
                "$nearSphere": [-118.09771, 33.89244]
            }
        },
        "scope": { "stash": {}, "maxLen": 10 },
        "finalize": function(key,value) {
            return { "msgs": stash[key] };                
        },
        "out": { "inline": 1 }
    }
)

This is a real cheat which just uses the "global scope" to keep a single object whose keys are the grouping keys. The results are pushed onto an array in that global object until the maximum length is reached. Results are already sorted by nearest, so the mapper just gives up doing anything with the current document after the 10 are reached per key.

The reducer wont be called since only 1 document per key is emitted. The finalize then just "pulls" the value from the global and returns it in the result.

Simple, but of course you don't have all the $geoNear options if you really need them, and this form has the hard limit of 100 document as the output from the initial query.