A queueing system for Perl

Question

I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs.

I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so jobs enqueued. What are good alternatives to this module which can be used in Perl?

Answer 1

I've had pretty good success with using ZeroMQ for this sort of problem, both with Perl and other languages.

In my experience, the ZeroMQ module appears to be the most reliable binding for Perl currently.

Answer 2

I haven't tried it under the conditions you list, but Thread::Queue has proven useful to me. Combined with forks, it can be used to communicate with processes, as long as those processes were spawned by the queue creator.

use forks;  # If you want to use processes instead of threads.
use Thread::Queue qw( );

A worker model is usually ideal.

my $q = Thread::Queue->new();

my @workers;
for (1..$NUM_WORKERS) {
   push @workers, async {
      while (my $item = $q->dequeue()) {
         ...
      }
   };
}

# ... Enqueue requests [ $q->enqueue($request); ] ...

# Signal termination
$q->enqueue(undef) for 1..@workers;

# Collect workers.
$_->join() for @workers;