python async post requests

问题

I was wondering if there was any way to make this script a lot faster - like instantly create 1000 accounts for example or at least in a matter of a few seconds. I’ve tried doing some async stuff myself but this is as far as I could get, I am just a beginner with asynchronous programming so any help is appreciated.

import asyncio
import aiohttp


async def make_numbers(numbers, _numbers):
    for i in range(numbers, _numbers):
        yield i

async def make_account():
   url = "https://example.com/sign_up.php"
   async with aiohttp.ClientSession() as session:
          async for x in make_numbers(35691, 5000000):
              async with  session.post(url, data ={
                    "terms": 1,
                    "captcha": 1,
                    "email": "user%s@hotmail.com" % str(x),
                    "full_name": "user%s" % str(x),
                    "password": "123456",
                    "username": "auser%s" % str(x)
              }) as response:
                    data = await response.text()
                    print("-> Creating account number %d" % x)
                    print (data)

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(make_account())
finally:
    loop.close()

回答1:

The code in the question executes all POST requests in a series, making the code no faster than if you used requests in a single thread. But unlike requests, asyncio makes it possible to parallelize them in the same thread:

async def make_account():
    url = "https://example.com/sign_up.php"
    async with aiohttp.ClientSession() as session:
        post_tasks = []
        # prepare the coroutines that post
        async for x in make_numbers(35691, 5000000):
            post_tasks.append(do_post(session, url, x))
        # now execute them all at once
        await asyncio.gather(*post_tasks)

async def do_post(session, url, x):
    async with session.post(url, data ={
                "terms": 1,
                "captcha": 1,
                "email": "user%s@hotmail.com" % str(x),
                "full_name": "user%s" % str(x),
                "password": "123456",
                "username": "auser%s" % str(x)
          }) as response:
          data = await response.text()
          print("-> Created account number %d" % x)
          print (data)

The above code will attempt to send all the POST requests at once. Despite the intention, it will be throttled by aiohttp.ClientSession's TCP connector which allows a maximum of 100 simultaneous connections by default. To increase or remove this limitation, you must set a custom connector on the session.

回答2:

The async code you posted looks ok to me. You can speed it up by combining asyncio with multithreading/multiprocess.

But there are other limitations that prevent you from creating 1000 accounts in a second. For example, network speed, concurrent connections, rate limit, database IOPS on the server side.

来源：https://stackoverflow.com/questions/51699817/python-async-post-requests

标签

python

python-requests

python-asyncio

aiohttp