How do I make 25 requests at a time with HTTP::Async in Perl?

Question

I\'m doing a lot of HTTP requests and I chose HTTP::Async to do the job. I\'ve over 1000 requests to make, and if I simply do the following (see code below), a lot of requests t

Answer 1

You're close, you just need to combine the two approaches! :-)

Untested, so think of it as pseudo code. In particular I am not sure if total_count is the right method to use, the documentation doesn't say. You could also just have an $active_requests counter that you ++ when adding a request and -- when you get a response.

while (1) {

   # if there aren't already 25 requests "active", then add more
   while (@urls and $async->total_count < 25) {
       my $url = shift @urls;
       $async->add( ... );
   }

   # deal with any finished requests right away, we wait for a
   # second just so we don't spin in the main loop too fast.
   while (my $response = $async->wait_for_next_response(1)) {
      # use $response
   }

   # finish the main loop when there's no more work
   last unless ($async->total_count or @urls);

}