bash: start multiple chained commands in background

Question

I\'m trying to run some commands in paralel, in background, using bash. Here\'s what I\'m trying to do:

forloop {
  //this part is actually written in perl
          


        

Answer 1

Forking in a for loop:

for i in x; do ((a; b; c;)&); done

Example:

for i in 500 300 100; do ((printf "Start $i: "; date; dd if=/dev/zero of=testfile_$i bs=1m count=$i 2>/dev/null; printf "End $i: "; date;)&) && sleep 1; done

Answer 2

The facility in bash that you're looking for is called Compound Commands. See the man page for more info:

Compound Commands A compound command is one of the following:

   (list) list  is  executed  in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below).  Variable assignments and
          builtin commands that affect the shell's environment do not remain in effect after  the  command  completes.   The
          return status is the exit status of list.

   { list; }
          list  is  simply  executed in the current shell environment.  list must be terminated with a newline or semicolon.
          This is known as a group command.  The return status is the exit status of list.  Note that unlike the metacharac‐
          ters  (  and  ),  {  and  } are reserved words and must occur where a reserved word is permitted to be recognized.
          Since they do not cause a word break, they must be separated from list by whitespace or another shell  metacharac‐
          ter.

There are others, but these are probably the 2 most common types. The first, the parens, will run a list of command in series in a subshell, while the second, the curly braces, will a list of commands in series in the current shell.

parens

% ( date; sleep 5; date; )
Sat Jan 26 06:52:46 EST 2013
Sat Jan 26 06:52:51 EST 2013

curly braces

% { date; sleep 5; date; }
Sat Jan 26 06:52:13 EST 2013
Sat Jan 26 06:52:18 EST 2013

Answer 3

Try to put commands in curly braces with &s, like this:

{command1 & ; command2 & ; command3 & ; }

This does not create a sub-shell, but executes the group of commands in the background.

HTH

Answer 4

Another way is to use the following syntax:

{ command1; command2; command3; } &
wait

Note that the & goes at the end of the command group, not after each command. The semicolon after the final command is necessary, as are the space after the first bracket and before the final bracket. The wait at the end ensures that the parent process is not killed before the spawned child process (the command group) ends.

You can also do fancy stuff like redirecting stderr and stdout:

{ command1; command2; command3; } 2>&2 1>&1 &

Your example would look like:

forloop() {
    { touch .file1.lock; cp bigfile1 /destination; rm .file1.lock; } &
}
# ... do some other concurrent stuff
wait # wait for childs to end

Answer 5

You can pass parameters to a command group (having sequential commands) and run them in background.

for hrNum in {00..11};
do
    oneHour=$((10#$hrNum + 0))
    secondHour=$((10#$hrNum + 12))
    { echo "$oneHour"; echo "$secondHour"; } &
    wait
done

Answer 6

You can use GNU parallel command to run jobs in parallel. It is more safe are faster.

My guess is that you are trying to copy multiple large files from source to destination. And for that you can do that in parallel with below statement.

$ ls *|parallel -kj0 --eta 'cp {} /tmp/destination'

As we have used -j0 option, all the files will be copied in parallel. In case if you need to reduce the number of parallel process then you can use -j<n> where <n> is the number of parallel process to be executed.

Parallel will also collect the output of the process and report it in a sequential manner (with -k option) which other job control mechanism cannot do.

--eta option will give you a details statistics of the process that is going on. So we can know how may of the process have been completed and how long will it take to get finished.