How can I run a list of commands in parallel?

Question

I have a file containing command lines that I want to run. This file contains around 2,000 lines.

I have 8 cores available. Is it possible to parse the file and start 8 processes, then execute another one from the file whenever one of the programs finishes? I want this to continue until the end of file is reached.

Answer 1

You can use xargs to read in the file, while limiting the maximum number of processes to the number of available cores. For example:

cores=$(fgrep -c processor /proc/cpuinfo)
xargs --arg-file=/tmp/foo \
      --max-procs=$cores  \
      --replace \
      --verbose \
      /bin/sh -c "{}"

Answer 2

Use GNU parallel. It's an incredibly powerful tool and official packages exist for about 20 or so linux distros. What's that? You have an excuse as to why you can't use it? Here's a simple example showing how to run a list or file of commands in parallel:

Contents of jobs.txt:

sleep 1; echo "a"
sleep 3; echo "b"
sleep 2; echo "c"

Command:

time parallel :::: jobs.txt

Results:

a
c
b

real    0m3.332s
user    0m0.170s
sys     0m0.037s

---

Notes:

If you wish to keep the order the same as the input, pass the -k flag to GNU parallel.

If you have more than eight cores and only wish to process with eight cores, add -j 8 to the args list.

The man page is a good read, but if you haven't already read this tutorial I would highly recommend the time investment.

Answer 3

You can start new processes on the background simply by running a command with &. There is an example here describing a solution of your problem.