PyMongo raises [errno 49] can't assign requested address after a large number of queries

Question

I have a MongoDB collection with > 1,000,000 documents. I am performing an initial .find({ my_query }) to return a subset of those documents (~25,

Answer 1

This is because you are using PyMongo incorrectly. You are creating a new MongoClient for each query, which requires you to open a new socket for each new query. This defeats PyMongo's connection pooling, and besides being extremely slow, it also means you open and close sockets faster than your TCP stack can keep up: you leave too many sockets in TIME_WAIT state so you eventually run out of ports.

Luckily, the fix is simple. Create one MongoClient and use it throughout:

conn = pymongo.MongoClient('mongodb://localhost:27017')
coll = conn.databases['race_results']

def _perform_queries(query):
    return coll.find(query).sort("date", -1)