gevent / requests hangs while making lots of head requests

Question

I need to make 100k head requests, and I\'m using gevent on top of requests. My code runs for a while, but then eventually hangs. I\'m not sure why it\'s hanging, or whether it\

Answer 1

I'm not sure if this will resolve your issue, but you are not using pool.Pool() correctly.

Try this:

def expand_short_urls(short_urls, chunk_size=100):
    # Pool() automatically limits your process to chunk_size greenlets running concurrently
    # thus you don't need to do all that chunking business you were doing in your for loop
    p = pool.Pool(chunk_size)
    print 'Expanding %d short_urls' % len(short_urls)

    # spawn() (both gevent.spawn() and Pool.spawn()) returns a gevent.Greenlet object
    # NOT the value your function, get_head, will return
    threads = [p.spawn(get_head, short_url) for short_url in short_urls]
    p.join()

    # to access the returned value of your function, access the Greenlet.value property
    results = {short_url: thread.value.url for short_url, thread in zip(short_urls, threads) 

if thread.value is not None and thread.value.status_code == 200} return results