Celery + Eventlet + non blocking requests

Question

I am using Python requests in celery workers to make large number of (~10/sec) API calls(includes GET,POST, PUT, DELETE). Each request takes around 5-10s to complete.

I tried running celery workers in eventlet pool, with 1000 concurrency.

Since requests are blocking process each concurrent connection is waiting on one request.

How do I make requests asynchronous?

Answer 1

Use eventlet monkey patching to make any pure python library non-blocking.

• patch single library

  # import requests  # instead do this:
  import eventlet
  requests = eventlet.import_patched('requests')
  

  packages erequests and grequests could be stripped down to these two lines.

• patch everything

  import eventlet
  eventlet.monkey_patch()  # must execute as early as possible
  ...
  # everything is non-blocking now:
  import requests, amqp, memcache, paramiko, redis
  

Update: there is known issue with monkey patching requests library. If you get:

ImportError: cannot import name utils

, then modify import line to

requests = eventlet.import_patched('requests.__init__')

Answer 2

from the docs:

there are lots of projects out there that combine Requests with one of Python’s asynchronicity frameworks. Two excellent examples are grequests and requests-futures.

for eventlet specifically you can use erequests.