Running a limited number of child processes in parallel in bash? [duplicate]

Answer 1

Another option:

PARALLEL_MAX=...
function start_job() {
  while [ $(ps --no-headers -o pid --ppid=$$ | wc -l) -gt $PARALLEL_MAX ]; do
    sleep .1  # Wait for background tasks to complete.                         
  done
  "$@" &
}
start_job some_big_command1
start_job some_big_command2
start_job some_big_command3
start_job some_big_command4
...

Answer 2

This code worked quite well for me.

I noticed one issue in which the script couldn't end. If you run into a case where the script wont end due to max_jobs being greater than the number of elements in the array, the script will never quit.

To prevent the above scenario, I've added the following right after the "max_jobs" declaration.

if [ $max_jobs -gt ${#todo_array[*]} ];
    then
           # there are more elements found in the array than max jobs, setting max jobs to #of array elements"
            max_jobs=${#todo_array[*]}
 fi

Answer 3

I know I'm late to the party with this answer but I thought I would post an alternative that, IMHO, makes the body of the script cleaner and simpler. (Clearly you can change the values 2 & 5 to be appropriate for your scenario.)

function max2 {
   while [ `jobs | wc -l` -ge 2 ]
   do
      sleep 5
   done
}

find . -type f | while read name ; 
do 
   max2; some_heavy_processing_command ${name} &
done
wait

Answer 4

Here is a very good function I used to control the maximum # of jobs from bash or ksh. NOTE: the - 1 in the pgrep subtracts the wc -l subprocess.

function jobmax
{
    typeset -i MAXJOBS=$1
    sleep .1
    while (( ($(pgrep -P $$ | wc -l) - 1) >= $MAXJOBS ))
    do
        sleep .1
    done
}

nproc=5
for i in {1..100}
do
    sleep 1 &
    jobmax $nproc
done
wait # Wait for the rest

Answer 5

I think I found a more handy solution using make:

#!/usr/bin/make -f

THIS := $(lastword $(MAKEFILE_LIST))
TARGETS := $(shell find . -name '*.sh' -type f)

.PHONY: all $(TARGETS)

all: $(TARGETS)

$(TARGETS):
        some_heavy_processing_command $@

$(THIS): ; # Avoid to try to remake this makefile

Call it as e.g. 'test.mak', and add execute rights. If You call ./test.mak it will call the some_heavy_processing_command one-by-one. But You can call as ./test.mak -j 4, then it will run four subprocesses at once. Also You can use it on a more sophisticated way: run as ./test.mak -j 5 -l 1.5, then it will run maximum 5 sub-processes while the system load is under 1.5, but it will limit the number of processes if the system load exceeds 1.5.

It is more flexible than xargs, and make is part of the standard distribution, not like parallel.

Answer 6

With GNU Parallel it becomes simpler:

find . -type f | parallel  some_heavy_processing_command {}

Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1