How to use both Python multiprocessing and asyncio?

心不动则不痛 提交于 2021-02-05 09:39:21

问题


import asyncio
import httpx
from datetime import datetime


async def request_test(url):
    async with httpx.AsyncClient() as client:
        r = await client.get(url, timeout=None, headers=None)
        return len(r.text)

async def main(rest_api_url_list ):
    futures = [asyncio.ensure_future(request_test(url)) for url in rest_api_url_list ]
    results = await asyncio.gather(*futures)
    print(results)
    print(len(results))


start = datetime.now()
rest_api_url_list = [~~~~~~~~~~~~~]    # 2000EA
loop = asyncio.get_event_loop()
loop.run_until_complete(main(rest_api_url_list ))
loop.close()
end = datetime.now()

Hi, I have 2000 api adress. And I need to call 2000 concurrently in one VM. So, I used the asyncio library to modify the code as above. But, This solution is not satisfactory. How can I increase the effect of parallel processing? I think I have to use multiprocessing and asyncio at the same time.

来源:https://stackoverflow.com/questions/64851881/how-to-use-both-python-multiprocessing-and-asyncio

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!