Shell Script for multithreading a process

Question

I am a Bioinformatician and recently stuck in a problem which requires some scripting to speed up my process. We have a software called PHASE and Command that i type in my c

Answer 1

"multi-threading" is the wrong word for what you are trying to do. You want to run multiple processes in parallel. Multi-threading refers to having multiple threads of execution running in the same process. Running all of the processes at once and letting the os schedule them for you has been mentioned, as has xargs -P, and you might want to look at gnu parallel. You can also hack a solution in the shell, but this has several issues (namely, it is not even remotely robust). The basic idea is to create a pipe and have each process write a token into the pipe when it is done. At the same time, you read the pipe and start up a new process whenever a token appears. For example:

#!/bin/bash 

n=${1-4}  # Use first arg as number of processes to run, default is 4

trap 'rm -vf /tmp/fifo' 0
rm -f /tmp/fifo
mkfifo /tmp/fifo

cmd() {
    ./PHASE test$1.inp test$1.out
    echo $1 > /tmp/fifo
}

# spawn first $n processes
yes | nl | sed ${n}q | while read num line; do
        cmd $num &
done


# Spawn a new process whenever a running process terminates
yes | nl | sed -e 1,${n}d -e 1000q | {
while read num line; do
        read -u 5 stub # wait for one to terminate
        cmd $num &
done 5< /tmp/fifo
wait
} &
exec 3> /tmp/fifo
wait