Please introduce a multi-processing library in Perl or Ruby

Question

In python we can use multiprocessing modules. If there is a similar library in Perl and Ruby, would you teach it? I would appreciate it if you can include a brief sample.

Answer 1

Have a look at this nice summary page for Perl parallel processing libs http://www.openfusion.net/perl/parallel_processing_perl_modules. I like Parallel::Forker, its a modern and more powerful library than the older Parallel::ForkManager and has more features like signalling child processes. I've used it in multiple projects and it works exactly as expected. Here's an example of how to use it:

#!/usr/bin/env perl

use strict;
use warnings;
use Parallel::Forker;

my $forker = Parallel::Forker->new(use_sig_child => 1, max_proc => 4);
$SIG{CHLD} = sub {
    Parallel::Forker::sig_child($forker);
};
$SIG{TERM} = sub {
    $forker->kill_tree_all('TERM') if $forker and $forker->in_parent;
    die "Exiting child process...\n";
};
# an example
for (1..10) {
    $forker->schedule(run_on_start => sub {
        # all child process code to run here
    })->ready();
}
# wait for all child processes to finish
$forker->wait_all();

Hope this helps you

Answer 2

Ruby:

• Working with multiple processes in Ruby
• Concurrency is a Myth in Ruby

Perl:

• Harnessing the power of multicore
• Why Perl Is a Great Language for Concurrent Programming

Also, Perl's threads are native operating system threads, so you can just use those to take advantage of multiple cores.

Answer 3

Check out Coro which provide coroutines to Perl.

Here is an excerpt from the authors docs....

This module collection manages continuations in general, most often in the form of cooperative threads (also called coros, or simply "coro" in the documentation). They are similar to kernel threads but don't (in general) run in parallel at the same time even on SMP machines. The specific flavor of thread offered by this module also guarantees you that it will not switch between threads unless necessary, at easily-identified points in your program, so locking and parallel access are rarely an issue, making thread programming much safer and easier than using other thread models.

Unlike the so-called "Perl threads" (which are not actually real threads but only the windows process emulation ported to unix, and as such act as processes), Coro provides a full shared address space, which makes communication between threads very easy. And Coro's threads are fast, too: disabling the Windows process emulation code in your perl and using Coro can easily result in a two to four times speed increase for your programs. A parallel matrix multiplication benchmark runs over 300 times faster on a single core than perl's pseudo-threads on a quad core using all four cores.

For something similar to above in Ruby then have a look at Fiber which comes with Ruby 1.9.

Here are two interesting articles on using Fiber:

• fibers cooperative scheduling in ruby
• pipelines using fibers

There is also a Fiber for Perl using Coro. Here are some articles about Fiber for Perl (in Japanese):

• blog post on writing Fiber for Perl using Coro
• Fiber for Perl slides

Answer 4

https://github.com/pmahoney/process_shared for ruby

Answer 5

With Perl, you have options. One option is to use processes as below. I need to look up how to write the analogous program using threads but http://perldoc.perl.org/perlthrtut.html should give you an idea.

#!/usr/bin/perl

use strict;
use warnings;

use Parallel::ForkManager;

my @data = (0 .. 19);

my $pm = Parallel::ForkManager->new(4);

for my $n ( @data ) {
    my $pid = $pm->start and next;
    warn sprintf "%d^3 = %d\n", $n, slow_cube($n);
    $pm->finish;
}

sub slow_cube {
    my ($n) = @_;

    sleep 1;
    return $n * $n * $n;
}

__END__

The following version using threads does not use a limit on the number of threads created (because I do not know how):

#!/usr/bin/perl

use strict;
use warnings;

use threads;

my @data = (0 .. 19);
my @threads = map { 
    threads->new( {context => 'list' }, \&slow_cube, $_ )
} @data;

for my $thr ( @threads ) {
    my ( $n, $ncubed ) = $thr->join;
    print "$n^3 = $ncubed\n";
}

sub slow_cube {
    my ($n) = @_;

    sleep 1;
    return $n, $n * $n * $n;
}

__END__

Interestingly:

TimeThis :  Command Line :  t.pl
TimeThis :  Elapsed Time :  00:00:01.281