Forking / Multi-Threaded Processes | Bash

Question

I would like to make a section of my code more efficient. I\'m thinking of making it fork off into multiple processes and have them execute 50/100 times at once, instead of

Answer 1

haridsv's approach is great, it gives the flexibility to run a processor slots setup where a number of processes can be kept running with new jobs submitting as jobs complete, keeping the overall load up. Here are my mods to haridsv's code for an n-slot processor for a 'grid' of ngrid 'jobs' ( I use it for grids of simulation models ) Followed by test output for 8 jobs 3 at a time, with running totals of running, submitted, completed and remaining

#!/bin/bash
########################################################################
# see haridsv on forking-multi-threaded-processes-bash
# loop over grid, submitting jobs in the background.
# As jobs complete new ones are set going to keep the number running
# up to n as much as possible, until it tapers off at the end.
#
# 8 jobs
ngrid=8
# 3 at a time
n=3
# running counts
running=0
completed=0
# previous values
prunning=0
pcompleted=0
#
########################################################################
# process monitoring functions
#
declare -a pids
#
function checkPids() {
echo  ${#pids[@]}
if [ ${#pids[@]} -ne 0 ]
then
    echo "Checking for pids: ${pids[@]}"
    local range=$(eval echo {0..$((${#pids[@]}-1))})
    local i
    for i in $range; do
        if ! kill -0 ${pids[$i]} 2> /dev/null; then
            echo "Done -- ${pids[$i]}"
            unset pids[$i]
            completed=$(expr $completed + 1)
        fi
    done
    pids=("${pids[@]}") # Expunge nulls created by unset.
    running=$((${#pids[@]}))
    echo "#PIDS :"$running
fi
}
#
function addPid() {
    desc=$1
    pid=$2
    echo " ${desc} - "$pid
    pids=(${pids[@]} $pid)
}
########################################################################
#
# Loop and report when job changes happen,
# keep going until all are completed.
#
idx=0
while [ $completed -lt ${ngrid} ]
do
#
    if [ $running -lt $n ] && [ $idx -lt ${ngrid} ]
    then
####################################################################
#
# submit a new process if less than n
# are running and we haven't finished...
#
# get desc for process
#
        name="job_"${idx}
# background execution
        sleep 3 &
        addPid $name $!
        idx=$(expr $idx + 1)
#
####################################################################
#
    fi
#
    checkPids
# if something changes...
    if [ ${running} -gt ${prunning} ] || \
       [ ${completed} -gt ${pcompleted} ]
    then
        remain=$(expr $ngrid - $completed)
        echo  " Running: "${running}" Submitted: "${idx}\
              " Completed: "$completed" Remaining: "$remain
    fi
# save counts to prev values
    prunning=${running}
    pcompleted=${completed}
#
    sleep 1
#
done
#
########################################################################

Test output:

 job_0 - 75257
1
Checking for pids: 75257
#PIDS :1
 Running: 1 Submitted: 1  Completed: 0 Remaining: 8
 job_1 - 75262
2
Checking for pids: 75257 75262
#PIDS :2
 Running: 2 Submitted: 2  Completed: 0 Remaining: 8
 job_2 - 75267
3
Checking for pids: 75257 75262 75267
#PIDS :3
 Running: 3 Submitted: 3  Completed: 0 Remaining: 8
3
Checking for pids: 75257 75262 75267
Done -- 75257
#PIDS :2
 Running: 2 Submitted: 3  Completed: 1 Remaining: 7
 job_3 - 75277
3
Checking for pids: 75262 75267 75277
Done -- 75262
#PIDS :2
 Running: 2 Submitted: 4  Completed: 2 Remaining: 6
 job_4 - 75283
3
Checking for pids: 75267 75277 75283
Done -- 75267
#PIDS :2
 Running: 2 Submitted: 5  Completed: 3 Remaining: 5
 job_5 - 75289
3
Checking for pids: 75277 75283 75289
#PIDS :3
 Running: 3 Submitted: 6  Completed: 3 Remaining: 5
3
Checking for pids: 75277 75283 75289
Done -- 75277
#PIDS :2
 Running: 2 Submitted: 6  Completed: 4 Remaining: 4
 job_6 - 75298
3
Checking for pids: 75283 75289 75298
Done -- 75283
#PIDS :2
 Running: 2 Submitted: 7  Completed: 5 Remaining: 3
 job_7 - 75304
3
Checking for pids: 75289 75298 75304
Done -- 75289
#PIDS :2
 Running: 2 Submitted: 8  Completed: 6 Remaining: 2
2
Checking for pids: 75298 75304
#PIDS :2
2
Checking for pids: 75298 75304
Done -- 75298
#PIDS :1
 Running: 1 Submitted: 8  Completed: 7 Remaining: 1
1
Checking for pids: 75304
Done -- 75304
#PIDS :0
 Running: 0 Submitted: 8  Completed: 8 Remaining: 0

Answer 2

In bash scripts (non-interactive) by default JOB CONTROL is disabled so you can't do the the commands: job, fg, and bg.

Here is what works well for me:

#!/bin/sh

set -m # Enable Job Control

for i in `seq 30`; do # start 30 jobs in parallel
  sleep 3 &
done

# Wait for all parallel jobs to finish
while [ 1 ]; do fg 2> /dev/null; [ $? == 1 ] && break; done

The last line uses "fg" to bring a background job into the foreground. It does this in a loop until fg returns 1 ($? == 1), which it does when there are no longer any more background jobs.