Execute multiple shell scripts concurrently

Question

I want to do the following things:

• Execute multiple shell scripts (here 2 scripts) concurrently.

• Wait until both scripts finish

Answer 1

The answer you seek is in this question shell - get exit code of background process

Basically, when you background a process you can't get its exit code directly. But if you run the bash wait command, then wait's exit code will return the exit code of the background process.

./a.sh &
pid1=$!
./b.sh
ret2=$?
wait ${pid1}
ret1=$?

This will work even if a.sh ends before you run wait. The special variable $? holds the exit code of the previous process. And $! holds the Process ID of the previously run process.

Answer 2

Backticks do not give the value returned by the command, but the output of the command. To get the return values:

#!/bin/sh

./a.sh &
./b.sh
ret2=$?   # get value returned by b.sh
wait %1   # Wait for a.sh to finish
ret1=$?   # get value returned by a.sh
echo "$ret1: $ret2"

If you mistated the question and do in fact want the output of the commands, you get that as well since they will both go to the stdout of the script.

Answer 3

Here is some code that I have been running, that seems to do exactly what you want. Just insert ./a.sh and ./b.sh where appropriate:

# Start the processes in parallel...
./script1.sh 1>/dev/null 2>&1 &
pid1=$!
./script2.sh 1>/dev/null 2>&1 &
pid2=$!
./script3.sh 1>/dev/null 2>&1 &
pid3=$!
./script4.sh 1>/dev/null 2>&1 &
pid4=$!

# Wait for processes to finish...
echo -ne "Commands sent... "
wait $pid1
err1=$?
wait $pid2
err2=$?
wait $pid3
err3=$?
wait $pid4
err4=$?

# Do something useful with the return codes...
if [ $err1 -eq 0 -a $err2 -eq 0 -a $err3 -eq 0 -a $err4 -eq 0 ]
then
    echo "pass"
else
    echo "fail"
fi

Note that this captures the exit status of the script and not what it outputs to stdout. There is no easy way of capturing the stdout of a script running in the background, so I would advise you to use the exit status to return information to the calling process.

Answer 4

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel -j0 '{}; echo $?' ::: a.sh b.sh

I have a suspicion that you want the exit code to check if one of them failed, and that you actually do not care what the precise exit code was. In that case you can do:

parallel -j0 ::: a.sh b.sh || echo one or both of them failed

If it is sufficient to get the error code of the last that failed:

parallel -j0 --halt 1 ::: a.sh b.sh; echo $?

Maybe you would like to kill a.sh if b.sh finishes early but fails:

parallel -j0 --halt 2 ::: a.sh b.sh; echo $?

You can install GNU Parallel simply by:

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 67bd7bc7dc20aff99eb8f1266574dadb
12345678 67bd7bc7 dc20aff9 9eb8f126 6574dadb
$ md5sum install.sh | grep b7a15cdbb07fb6e11b0338577bc1780f
b7a15cdb b07fb6e1 1b033857 7bc1780f
$ sha512sum install.sh | grep 186000b62b66969d7506ca4f885e0c80e02a22444
6f25960b d4b90cf6 ba5b76de c1acdf39 f3d24249 72930394 a4164351 93a7668d
21ff9839 6f920be5 186000b6 2b66969d 7506ca4f 885e0c80 e02a2244 40e8a43f
$ bash install.sh

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Print the cheat sheet: https://www.gnu.org/software/parallel/parallel_cheat.pdf

Answer 5

If you have bash 4.2 or later available the following might be useful to you. It uses associative arrays to store task names and their "code" as well as task names and their pids. I have also built in a simple rate-limiting method which might come handy if your tasks consume a lot of CPU or I/O time and you want to limit the number of concurrent tasks.

The script launches all tasks in the first loop and consumes the results in the second one.

This is a bit overkill for simple cases but it allows for pretty neat stuff. For example one can store error messages for each task in another associative array and print them after everything has settled down.

(I have copied this answer over from my answer here because it solves both questions, if that's not ok please tell me or replace it directly with just a link or whatever is suitable.)

#! /bin/bash

main () {
    local -A pids=()
    local -A tasks=([task1]="echo 1"
                    [task2]="echo 2"
                    [task3]="echo 3"
                    [task4]="false"
                    [task5]="echo 5"
                    [task6]="false")
    local max_concurrent_tasks=2

    for key in "${!tasks[@]}"; do
        while [ $(jobs 2>&1 | grep -c Running) -ge "$max_concurrent_tasks" ]; do
            sleep 1 # gnu sleep allows floating point here...
        done
        ${tasks[$key]} &
        pids+=(["$key"]="$!")
    done

    errors=0
    for key in "${!tasks[@]}"; do
        pid=${pids[$key]}
        local cur_ret=0
        if [ -z "$pid" ]; then
            echo "No Job ID known for the $key process" # should never happen
            cur_ret=1
        else
            wait $pid
            cur_ret=$?
        fi
        if [ "$cur_ret" -ne 0 ]; then
            errors=$(($errors + 1))
            echo "$key (${tasks[$key]}) failed."
        fi
    done

    return $errors
}

main