Running Multiple Tasks in Parallel

Question

I have a list of proxies, each proxy goes to various sites and pulls the needed data from the sites. Currently it\'s doing this one at a time. But I\'d like to have 10 - 20 task

Answer 1

PArallel.ForEach does not work well as it expects a synchronous lambda and giving it an asynchronous one basically causes it to return as soon as it starts. There is a way around it though, check this question out: Is it OK to do some async/await inside some .NET Parallel.ForEach() code?

Answer 2

One way is to avoid awaiting in the foreach. The thing is that your await effectively blocks your execution. A better way might be something like this:

await Task.WhenAll(proxies.Select(p => DownloadDataFromSite(site)));

This will mean you'll be awaiting all the tasks at once, which means the asynchronous I/O is going to happen in parallel. Note that if you're doing CPU work too, that's not going to really be parallelized.

The point is, asynchronous I/O (such as downloading a web page) doesn't require more threads to run in parallel. On the other hand, Parallel.ForEach is actually intended for CPU-bound work, rather than I/O bound work, and it does execute the code on multiple threads.