How do I make 25 requests at a time with HTTP::Async in Perl?

Question

I\'m doing a lot of HTTP requests and I chose HTTP::Async to do the job. I\'ve over 1000 requests to make, and if I simply do the following (see code below), a lot of requests t

Answer 1

If you can't call wait_for_next_response fast enough because you're in the middle of executing other code, the simplest solution is to make the code interruptable by moving it to a separate thread of execution. But if you're going to start using threads, why use HTTP::Async?

use threads;
use Thread::Queue::Any 1.03;

use constant NUM_WORKERS => 25;

my $req_q = Thread::Queue::Any->new();
my $res_q = Thread::Queue::Any->new();

my @workers;
for (1..NUM_WORKERS) {
   push @workers, async {
      my $ua = LWP::UserAgent->new();
      while (my $req = $req_q->dequeue()) {
         $res_q->enqueue( $ua->request($req) );
      }
   };    
}

for my $url (@urls) {
   $req_q->enqueue( HTTP::Request->new( GET => $url ) );
}

$req_q->enqueue(undef) for @workers;

for (1..@urls) {
   my $res = $res_q->dequeue();
   ...
}

$_->join() for @workers;