Python aiohttp/asyncio - how to process returned data

Question

Im in the process of moving some synchronous code to asyncio using aiohttp. the synchronous code was taking 15 minutes to run, so I\'m hoping to improves this.

I have so

Answer 1

Your code isn't far from the mark. asyncio.gather returns the results in the order of the arguments, so order is preserved here, but page_content will not be called in order.

A few tweaks:

First of all, you do not need ensure_future here. Creating a Task is only needed if you are trying to have a coroutine outlive its parent, ie if the task has to continue running even though the function that created it is done. Here what you need is instead calling asyncio.gather directly with your coroutines:

async def get_url_data(urls, username, password):
    async with aiohttp.ClientSession(...) as session:
        responses = await asyncio.gather(*(fetch(session, i) for i in urls))
    for i in responses:
        print(i.title.text)
    return responses

But calling this would schedule all the fetch at the same time, and with a high number of URLs, this is far from optimal. Instead you should choose a maximum concurrency and ensure at most X fetches are running at any time. To implement this, you can use a asyncio.Semaphore(20), this semaphore can only be acquired by at most 20 coroutines, so the others will wait to acquire until a spot is available.

CONCURRENCY = 20
TIMEOUT = 15

async def fetch(session, sem, url):
    async with sem:
        async with session.get(url) as response:
            return page_content(await response.text())

async def get_url_data(urls, username, password):
    sem = asyncio.Semaphore(CONCURRENCY)
    async with aiohttp.ClientSession(...) as session:
        responses = await asyncio.gather(*(
            asyncio.wait_for(fetch(session, sem, i), TIMEOUT)
            for i in urls
        ))
    for i in responses:
        print(i.title.text)
    return responses

This way, all the fetches are started immediately, but only 20 of them will be able to acquire the semaphore. The others will block at the first async with instruction and wait until another fetch is done.

I have also replaced the aiohttp.Timeout with the official asyncio equivalent here.

Finally, for the actual processing of the data, if you are limited by CPU time, asyncio will probably not help you much. You will need to use a ProcessPoolExecutor here to parallelise the actual work to another CPU. run_in_executor will probably be of use to.