How do you run multiple programs in parallel from a bash script?

Question

I am trying to write a .sh file that runs many programs simultaneously

I tried this

prog1 
prog2

Answer 1

If you're:

• On Mac and have iTerm
• Want to start various processes that stay open long-term / until Ctrl+C
• Want to be able to easily see the output from each process
• Want to be able to easily stop a specific process with Ctrl+C

One option is scripting the terminal itself if your use case is more app monitoring / management.

For example I recently did the following. Granted it's Mac specific, iTerm specific, and relies on a deprecated Apple Script API (iTerm has a newer Python option). It doesn't win any elegance awards but gets the job done.

#!/bin/sh
root_path="~/root-path"
auth_api_script="$root_path/auth-path/auth-script.sh"
admin_api_proj="$root_path/admin-path/admin.csproj"
agent_proj="$root_path/agent-path/agent.csproj"
dashboard_path="$root_path/dashboard-web"

osascript <<THEEND
tell application "iTerm"
  set newWindow to (create window with default profile)

  tell current session of newWindow
    set name to "Auth API"
    write text "pushd $root_path && $auth_api_script"
  end tell

  tell newWindow
    set newTab to (create tab with default profile)
    tell current session of newTab
        set name to "Admin API"
        write text "dotnet run --debug -p $admin_api_proj"
    end tell
  end tell

  tell newWindow
    set newTab to (create tab with default profile)
    tell current session of newTab
        set name to "Agent"
        write text "dotnet run --debug -p $agent_proj"
    end tell
  end tell

  tell newWindow
    set newTab to (create tab with default profile)
    tell current session of newTab
        set name to "Dashboard"
        write text "pushd $dashboard_path; ng serve -o"
    end tell
  end tell

end tell
THEEND

Answer 2

With bashj ( https://sourceforge.net/projects/bashj/ ) , you should be able to run not only multiple processes (the way others suggested) but also multiple Threads in one JVM controlled from your script. But of course this requires a java JDK. Threads consume less resource than processes.

Here is a working code:

#!/usr/bin/bashj

#!java

public static int cnt=0;

private static void loop() {u.p("java says cnt= "+(cnt++));u.sleep(1.0);}

public static void startThread()
{(new Thread(() ->  {while (true) {loop();}})).start();}

#!bashj

j.startThread()

while [ j.cnt -lt 4 ]
do
  echo "bash views cnt=" j.cnt
  sleep 0.5
done

Answer 3

I had a similar situation recently where I needed to run multiple programs at the same time, redirect their outputs to separated log files and wait for them to finish and I ended up with something like that:

#!/bin/bash

# Add the full path processes to run to the array
PROCESSES_TO_RUN=("/home/joao/Code/test/prog_1/prog1" \
                  "/home/joao/Code/test/prog_2/prog2")
# You can keep adding processes to the array...

for i in ${PROCESSES_TO_RUN[@]}; do
    ${i%/*}/./${i##*/} > ${i}.log 2>&1 &
    # ${i%/*} -> Get folder name until the /
    # ${i##*/} -> Get the filename after the /
done

# Wait for the processes to finish
wait

Source: http://joaoperibeiro.com/execute-multiple-programs-and-redirect-their-outputs-linux/

Answer 4

You can use wait:

some_command &
P1=$!
other_command &
P2=$!
wait $P1 $P2

It assigns the background program PIDs to variables ($! is the last launched process' PID), then the wait command waits for them. It is nice because if you kill the script, it kills the processes too!

Answer 5

xargs -P <n> allows you to run <n> commands in parallel.

While -P is a nonstandard option, both the GNU (Linux) and macOS/BSD implementations support it.

The following example:

• runs at most 3 commands in parallel at a time,
• with additional commands starting only when a previously launched process terminates.

time xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF

The output looks something like:

1   # output from 1st command 
4   # output from *last* command, which started as soon as the count dropped below 3
2   # output from 2nd command
3   # output from 3rd command

real    0m3.012s
user    0m0.011s
sys 0m0.008s

The timing shows that the commands were run in parallel (the last command was launched only after the first of the original 3 terminated, but executed very quickly).

The xargs command itself won't return until all commands have finished, but you can execute it in the background by terminating it with control operator & and then using the wait builtin to wait for the entire xargs command to finish.

{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &

# Script execution continues here while `xargs` is running 
# in the background.
echo "Waiting for commands to finish..."

# Wait for `xargs` to finish, via special variable $!, which contains
# the PID of the most recently started background process.
wait $!

Note:

• BSD/macOS xargs requires you to specify the count of commands to run in parallel explicitly, whereas GNU xargs allows you to specify -P 0 to run as many as possible in parallel.

• Output from the processes run in parallel arrives as it is being generated, so it will be unpredictably interleaved.

  • GNU parallel, as mentioned in Ole's answer (does not come standard with most platforms), conveniently serializes (groups) the output on a per-process basis and offers many more advanced features.

Answer 6

There is a very useful program that calls nohup.

     nohup - run a command immune to hangups, with output to a non-tty