View progress of long running mongodb aggregation job

Question

I have a long running job using Mongodb\'s (2.6.0-rc2) aggregation framework: http://docs.mongodb.org/manual/core/aggregation-introduction/

I have written t

Answer 1

If you use the $out aggregation pipeline operator to output the result of the aggregation to another (or the same) collection, you can open a new mongo shell and see how many documents are in the new collection. If you're overwriting the collection you're aggregating from, MongoDB will use a temporary collection name in order to make the operation atomic, like tmp.agg_out.1. So, run

db['tmp.agg_out.1'].count()

To find out the exact name of the temporary collection, you can tail the current MongoDB log and watch for messages about the aggregation. mLab and other cloud MongoDB hosting providers may have a handy "stream current log" option as well.

For example, while running the query in this answer, the relevant log messages may look like this:

2019-04-05T03:55:42.126-0700 I COMMAND [conn244209] command collection.tmp.agg_out.1 appName: "MongoDB Shell" command: insert { insert: "tmp.agg_out.1", ordered: true, $db: "mydb" } ninserted:18145 keysInserted:351002 numYields:0 locks:{ Global: { acquireCount: { r: 70917, w: 61737 } }, Database: { ... }, Collection: { ... }, Metadata: { ... }, oplog: { ... } protocol:op_msg 161451ms

(I was hoping that nInserted or keysInserted would indicate progress, but that doesn't seem to be the case; the count of the documents in the temporary collection was a much more accurate progress indicator.)