In what way is grequests asynchronous?

Question

I\'ve been using the python requests library for some time, and recently had a need to make a request asynchronously, meaning I would like to send off the HTTP request, have

Answer 1

If you don't want to use grequests you can just implement requests with callbacks using requests + the threading module from the standard library. It's actually really easy, and if all you want to do is send requests with callbacks the API is nicer than the one provided by grequests.

from threading import Thread

from requests import get, post, put, patch, delete, options, head



request_methods = {
    'get': get,
    'post': post,
    'put': put,
    'patch': patch,
    'delete': delete,
    'options': options,
    'head': head,
}


def async_request(method, *args, callback=None, timeout=15, **kwargs):
    """Makes request on a different thread, and optionally passes response to a
    `callback` function when request returns.
    """
    method = request_methods[method.lower()]
    if callback:
        def callback_with_args(response, *args, **kwargs):
            callback(response)
        kwargs['hooks'] = {'response': callback_with_args}
    kwargs['timeout'] = timeout
    thread = Thread(target=method, args=args, kwargs=kwargs)
    thread.start()

You can verify that it works like an AJAX calls in JS: you send a request on another thread, do some stuff on the main thread, and when the request returns you invoke a callback. This callback just prints out the response content.

async_request('get', 'http://httpbin.org/anything', callback=lambda r: print(r.json()))
for i in range(10):
    print(i)

Answer 2

.map() is meant to run retrieval of several URLs in parallel, and will indeed wait for these tasks to complete (gevent.joinall(jobs)) is called).

Use .send() instead to spawn jobs, using a Pool instance:

req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))
job = grequests.send(req, grequests.Pool(1))

for i in range(10):
    print i

Without the pool the .send() call will block still, but only for the gevent.spawn() call it executes.

Answer 3

Create a list of requests and then send them with .imap:

event_list = [grequests.get(url_viol_card, params={"viol": i},
              session=session) for i in print_ev_list]
for r in grequests.imap(event_list, size=5):
    print(r.request.url)

• session is requests.Session() object (optional)
• size=5 send 5 requests simultaneously: as soon as one of them is completed the next one is sent
• In this example when any request (unordered) is completed its url is printed