Shell, run four processes parallel

Question

Currently I\'m stuck at running time consuming simulations efficiently. The intention is to run 4 simulations in parallel, because it\'s a single thread application and a qu

Answer 1

If you don't want to install the parallel utility (which looks neat assuming it works as indicated), then you could adapt this Perl script (basically, changing the command that is executed), and probably reducing the monitoring:

#!/usr/bin/env perl
use strict;
use warnings;
use constant MAX_KIDS => 4;

$| = 1;

my %pids;
my $kids = 0;   # Number of kids

for my $i (1..20)
{
    my $pid;
    if (($pid = fork()) == 0)
    {
        my $tm = int(rand() * (10 - 2) + 2);
        print "sleep $tm\n";
        # Using exec in a block on its own is the documented way to
        # avoid the warning:
        # Statement unlikely to be reached at filename.pl line NN.
        #   (Maybe you meant system() when you said exec()?)
        # Yes, I know the print and exit statements should never be
        # reached, but, dammit, sometimes things go wrong!
        { exec "sleep", $tm; }
        print STDERR "Oops: couldn't sleep $tm!\n";
        exit 1;
    }
    $pids{$pid} = 1;
    $kids++;
    my $time = time;
    print "PID: $pid; Kids: $kids; Time: $time\n";
    if ($kids >= MAX_KIDS)
    {
        my $kid = waitpid(-1, 0);
        print "Kid: $kid ($?)\n";
        if ($kid != -1)
        {
            delete $pids{$kid};
            $kids--;
        }
    }
}

while ((my $kid = waitpid(-1, 0)) > 0)
{
    my $time = time;
    print "Kid: $kid (Status: $?); Time: $time\n";
    delete $pids{$kid};
    $kids--;
}

# This should not do anything - and doesn't (any more!).
foreach my $pid (keys %pids)
{
    printf "Undead: $pid\n";
}

Example output:

PID: 20152; Kids: 1; Time: 1383436882
PID: 20153; Kids: 2; Time: 1383436882
sleep 5
PID: 20154; Kids: 3; Time: 1383436882
sleep 7
sleep 9
PID: 20155; Kids: 4; Time: 1383436882
sleep 4
Kid: 20155 (0)
PID: 20156; Kids: 4; Time: 1383436886
sleep 4
Kid: 20152 (0)
PID: 20157; Kids: 4; Time: 1383436887
sleep 2
Kid: 20153 (0)
PID: 20158; Kids: 4; Time: 1383436889
sleep 9
Kid: 20157 (0)
PID: 20159; Kids: 4; Time: 1383436889
sleep 6
Kid: 20156 (0)
PID: 20160; Kids: 4; Time: 1383436890
sleep 6
Kid: 20154 (0)
PID: 20161; Kids: 4; Time: 1383436891
sleep 9
Kid: 20159 (0)
PID: 20162; Kids: 4; Time: 1383436895
sleep 7
Kid: 20160 (0)
PID: 20163; Kids: 4; Time: 1383436896
sleep 9
Kid: 20158 (0)
PID: 20164; Kids: 4; Time: 1383436898
sleep 6
Kid: 20161 (0)
PID: 20165; Kids: 4; Time: 1383436900
sleep 9
Kid: 20162 (0)
PID: 20166; Kids: 4; Time: 1383436902
sleep 9
Kid: 20164 (0)
PID: 20167; Kids: 4; Time: 1383436904
sleep 2
Kid: 20163 (0)
PID: 20168; Kids: 4; Time: 1383436905
sleep 6
Kid: 20167 (0)
PID: 20169; Kids: 4; Time: 1383436906
sleep 9
Kid: 20165 (0)
PID: 20170; Kids: 4; Time: 1383436909
sleep 4
Kid: 20168 (0)
PID: 20171; Kids: 4; Time: 1383436911
Kid: 20166 (0)
sleep 9
Kid: 20170 (Status: 0); Time: 1383436913
Kid: 20169 (Status: 0); Time: 1383436915
Kid: 20171 (Status: 0); Time: 1383436920

Answer 2

NUMJOBS=30
NUMPOOLS=4

seq 1 "$NUMJOBS" | for p in $(seq 1 $NUMPOOLS); do
    while read x; do ./sim -r "$x"; done &
done

The for loop creates a pool of background processes which reads from the shared standard input to start a simulation. Each background process "blocks" while its simulation is running, then reads the next job number from the seq command.

Without the for loop, it might be a little easier to follow:

seq 1 "$NUMJOBS" | {
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
}

Assuming sim takes a non-trivial amount of time to run, the first while will read 1 from its standard input, the 2nd 2, etc. Whichever sim finishes first, that while loop will read 5 from standard input; the next to finish will read 6, and so on. Once the last simulation is started, each read will fail, causing the loop to exit.

Answer 3

Install the moreutils package in Ubuntu, then use the parallel utility:

parallel -j 4 ./sim -r -- 1 2 3 4 5 6 7 8 ...