Forking / Multi-Threaded Processes | Bash

Question

I would like to make a section of my code more efficient. I\'m thinking of making it fork off into multiple processes and have them execute 50/100 times at once, instead of

Answer 1

Let me try example

for x in 1 2 3 ; do { echo a $x ; sleep 1 ; echo b $x ; } &  done ; sleep 10

And use jobs to see what's running.

Answer 2

Based on what you all shared I was able to put this together:

#!/usr/bin/env bash

VAR1="192.168.1.20 192.168.1.126 192.168.1.36"

for a in $VAR1; do { ssh -t -t $a -l Administrator "sudo softwareupdate -l"; } & done;
WAITPIDS="$WAITPIDS "$!;...; wait $WAITPIDS
echo "Script has finished"

Exit 1

This lists all the updates on the mac on three machines at once. Later on I used it to perform a software update for all machines when i CAT my ipaddress.txt

Answer 3

Here's my thread control function:

#!/bin/bash
# This function just checks jobs in background, don't do more things.
# if jobs number is lower than MAX, then return to get more jobs;
# if jobs number is greater or equal to MAX, then wait, until someone finished.

# Usage:
#   thread_max 8
#   thread_max 0    # wait, until all jobs completed

thread_max() {
    local CHECK_INTERVAL="3s"
    local CUR_THREADS=
    local MAX=
    [[ $1 ]] && MAX=$1 || return 127

    # reset MAX value, 0 is easy to remember
    [ $MAX -eq 0 ] && {
        MAX=1
        DEBUG "waiting for all tasks finish"
    }

    while true; do
        CUR_THREADS=`jobs -p | wc -w`

        # workaround about jobs bug. If don't execute it explicitily,
        # CUR_THREADS will stick at 1, even no jobs running anymore.
        jobs &>/dev/null

        DEBUG "current thread amount: $CUR_THREADS"
        if [ $CUR_THREADS -ge $MAX ]; then
            sleep $CHECK_INTERVAL
        else
            return 0
        fi
    done
}

Answer 4

With GNU Parallel you can do:

cat file | parallel 'foo {}; foo2 {}; foo3 {}'

This will run one job on each cpu core. To run 50 do:

cat file | parallel -j 50 'foo {}; foo2 {}; foo3 {}'

Watch the intro videos to learn more:

http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 5

I don't know of any explicit fork call in bash. What you probably want to do is append & to a command that you want to run in the background. You can also use & on functions that you define within a bash script:

do_something_with_line()
{
  line=$1
  foo
  foo2
  foo3
}

for line in file
do
  do_something_with_line $line &
done

EDIT: to put a limit on the number of simultaneous background processes, you could try something like this:

for line in file
do
  while [`jobs | wc -l` -ge 50 ]
  do
    sleep 5
  done
  do_something_with_line $line &
done

Answer 6

I don't like using wait because it gets blocked until the process exits, which is not ideal when there are multiple process to wait on as I can't get a status update until the current process is done. I prefer to use a combination of kill -0 and sleep to this.

Given an array of pids to wait on, I use the below waitPids() function to get a continuous feedback on what pids are still pending to finish.

declare -a pids
waitPids() {
    while [ ${#pids[@]} -ne 0 ]; do
        echo "Waiting for pids: ${pids[@]}"
        local range=$(eval echo {0..$((${#pids[@]}-1))})
        local i
        for i in $range; do
            if ! kill -0 ${pids[$i]} 2> /dev/null; then
                echo "Done -- ${pids[$i]}"
                unset pids[$i]
            fi
        done
        pids=("${pids[@]}") # Expunge nulls created by unset.
        sleep 1
    done
    echo "Done!"
}

When I start a process in the background, I add its pid immediately to the pids array by using this below utility function:

addPid() {
    local desc=$1
    local pid=$2
    echo "$desc -- $pid"
    pids=(${pids[@]} $pid)
}

Here is a sample that shows how to use:

for i in {2..5}; do
    sleep $i &
    addPid "Sleep for $i" $!
done
waitPids

And here is how the feedback looks:

Sleep for 2 -- 36271
Sleep for 3 -- 36272
Sleep for 4 -- 36273
Sleep for 5 -- 36274
Waiting for pids: 36271 36272 36273 36274
Waiting for pids: 36271 36272 36273 36274
Waiting for pids: 36271 36272 36273 36274
Done -- 36271
Waiting for pids: 36272 36273 36274
Done -- 36272
Waiting for pids: 36273 36274
Done -- 36273
Waiting for pids: 36274
Done -- 36274
Done!