C# Running many async tasks the same time

Question

I\'m kinda new to async tasks.

I\'ve a function that takes student ID and scrapes data from specific university website with the required ID.

    private         


        

Answer 1

You are basically creating a denial of service attack on the website when you are queuing up 9000 requests in such a short time frame. Not only is this causing you errors, but it could take down the website. It would be best to limit the number of concurrent requests to a more reasonable value (say 30). While there are probably several ways to do this, one that comes to mind is the following:

private async Task Test()
{
  var tasks = new List();
  for (int i = ids.first; i <= ids.last; i++)
  {
    tasks.Add(/* Do stuff */);
    await WaitList(tasks, 30);
  }
}

private async Task WaitList(IList tasks, int maxSize)
{
  while (tasks.Count > maxSize)
  {
    var completed = await Task.WhenAny(tasks).ConfigureAwait(false);
    tasks.Remove(completed);
  }
}

Other approaches might leverage the producer/consumer pattern using .Net classes such as a BlockingCollection