MongoDB Query for records with non-existant field & indexing

Question

We have a mongo database with around 1M documents, and we want to poll this database using a processed field to find documents which we havent seen before. To do this we are

Answer 1

Its slow because checking for _processed -> not exists doesnt offer much selectivity. Its like having an index on "Gender" - and since there are only two possible options male or female then if you have 1M rows and an index on Gender it will have to scan 50% or 500K rows to find all males.

You need to make your index more selective.

Answer 2

Upgrading to 2.0 is going to do this for you:

From MongoDB.org:

Before v2.0, $exists is not able to use an index. Indexes on other fields are still used.