C# Running many async tasks the same time

Question

I\'m kinda new to async tasks.

I\'ve a function that takes student ID and scrapes data from specific university website with the required ID.

    private         


        

Answer 1

This is what I ended up with based on @erdomke code:

    public static async Task ForEachParallel<T>(
      this IEnumerable<T> list, 
      Func<T, Task> action, 
      int dop)
    {
        var tasks = new List<Task>(dop);
        foreach (var item in list)
        {
            tasks.Add(action(item));

            while (tasks.Count >= dop)
            {
                var completed = await Task.WhenAny(tasks).ConfigureAwait(false);
                tasks.Remove(completed);
            }
        }

        // Wait for all remaining tasks.
        await Task.WhenAll(tasks).ConfigureAwait(false);
    }

    // usage
    await Enumerable
        .Range(1, 500)
        .ForEachParallel(i => ProcessItem(i), Environment.ProcessorCount);

Answer 2

You are basically creating a denial of service attack on the website when you are queuing up 9000 requests in such a short time frame. Not only is this causing you errors, but it could take down the website. It would be best to limit the number of concurrent requests to a more reasonable value (say 30). While there are probably several ways to do this, one that comes to mind is the following:

private async Task Test()
{
  var tasks = new List<Task>();
  for (int i = ids.first; i <= ids.last; i++)
  {
    tasks.Add(/* Do stuff */);
    await WaitList(tasks, 30);
  }
}

private async Task WaitList(IList<Task> tasks, int maxSize)
{
  while (tasks.Count > maxSize)
  {
    var completed = await Task.WhenAny(tasks).ConfigureAwait(false);
    tasks.Remove(completed);
  }
}

Other approaches might leverage the producer/consumer pattern using .Net classes such as a BlockingCollection