In Perl, how can I wait for threads to end in parallel?

Question

I have a Perl script that launches 2 threads,one for each processor. I need it to wait for a thread to end, if one thread ends a new one is spawned. It seems that the join m

Answer 1

I think you need to move the code that pulls the next file from the list into the threads themselves.

So every thread would not just process one file, but continue to process until the list is empty.

This way, you also save on the overhead of creating new threads all the time.

Your main thread will then join both of them.

Of course, this requires synchronization on the list (so that they do not pull the same data). Alternately, you could split the list into two (one for each thread), but that might result in an unlucky distribution.

(PS: No Perl God, just a humble monk)

Answer 2

First off, a few comments on the code itself. You need to make sure you have:

use strict;
use warnings;

at the beginning of every script. Second:

@file_list = @ARGV;      #Our file list
$nofiles = $#file_list + 1; #Real number of files 

is unnecessary as an array in scalar context evaluates to the number of elements in the array. That is:

$nofiles = @ARGV;

would correctly give you the number of files in @ARGV regardless of the value of $[.

Finally, the script can be made much simpler by partitioning the list of files before starting the threads:

use strict; use warnings;

use threads;
use threads::shared;

my @threads = (
    threads->new(\&process, @ARGV[0 .. @ARGV/2]),
    threads->new(\&process, @ARGV[@ARGV/2 + 1 .. @ARGV - 1]),
);

$_->join for @threads;

sub process {
    my @files = @_;
    warn "called with @files\n";
    for my $file ( @files ) {
        warn "opening '$file'\n";
        sleep rand 3;
        warn "closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
called with 1 2 3
opening '1'
called with 4 5
opening '4'
closing '4'
opening '5'
closing '1'
opening '2'
closing '5'
closing '2'
opening '3'
closing '3'

Alternatively, you can let the threads move on to the next task as they are done:

use strict; use warnings;

use threads;
use threads::shared;

my $current :shared;
$current = 0;

my @threads = map { threads->new(\&process, $_) } 1 .. 2;
$_->join for @threads;

sub process {
    my ($thr) = @_;
    warn "thread $thr stared\n";

    while ( 1 ) {
        my $file;
        {
            lock $current;
            return unless $current < @ARGV;
            $file = $ARGV[$current];
            ++ $current;
        }
        warn "$thr: opening '$file'\n";
        sleep rand 5;
        warn "$thr: closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
thread 1 stared
1: opening '1'
1: closing '1'
1: opening '2'
thread 2 stared
2: opening '3'
2: closing '3'
2: opening '4'
1: closing '2'
1: opening '5'
1: closing '5'
2: closing '4'