Understanding requests versus grequests

Question

I\'m working with a process which is basically as follows:

1. Take some list of urls.
2. Get a Response object from each.
3. Create a BeautifulSoup object

Answer 1

You are just sending requests too fast. As grequests is an async lib, all of these requests are almost sent simultaneously. They are too many.

You just need to limit the concurrent tasks by grequests.map(rs, size=your_choice), I have tested grequests.map(rs, size=10) and it works well.