Parallel Processing in While Loop

问题

I have a .sh that is reading in lines from a .txt to be used as parameters for another shell script.Every two lines process in parallel. Currently: My current code will read in ALL lines in the file (2 at a time) for the first .sh call, then ALL lines for second .sh call, then ALL for the last .sh call

Problem: I need to first two lines in first .sh, then second .sh, then last .sh..THEN loop back and process with next two lines HHEEELLLPPP!!! :)

Now:

cat XXXXX.txt | while read line; do
export step=${line//\"/}
export step=ExecuteModel_${step//,/_}
export pov=$line

$dir"/hpm_ws_client.sh" processCalcScriptOptions  "$appName" "$pov" "$layers" "$stages" "" "$stages" "$stages" FALSE > "$dir""/"$step"_ProcessID.log"
$dir_shell_model"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &


$dir"/hpm_ws_client.sh" processCalcScriptOptions "$appName" "$pov" "$layers" "" ""  "$stage_final" "" TRUE > "$dir""/""$step""_ProcessID.log"
$dir"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &

$dir"/hpm_ws_client.sh" processGenealogyExecutionPaths "$appName" "$pov" "$layers" "$genPath_01" > "$dir""/""$step""_ProcessID.log"
$dir"/check_process_status.sh" "$dir" "$step" > "$dir""/""$step""_Monitor.log" &

if (( ++i % 2 == 0))
then
echo "Waiting..."
wait
fi
done

回答1:

I cannot see what you are really trying to do, but hope one of these two syntaxes will help - either reading two lines at a time, or loading the parameters into an array an re-using them.

So, if your file.txt looks like this:

line 1
line 2
line 3
line 4
line 5
line 6

Example 1 - with two reads

#!/bin/bash
while read a && read b; do
   echo $a, $b
done < file.txt

Output

line 1, line 2
line 3, line 4
line 5, line 6

Example 2 - with a bash array

#!/bin/bash
declare -a params
while IFS=$'\n' read -r z; do
    params+=("${z}")
done < file.txt

# Now print the elements out
for (( i=0;i<${#params[@]};i++ )) do
   echo ${params[$i]}
done

Output

line 1
line 2
line 3
line 4
line 5
line 6

Example 3 - with GNU Parallel

Or, as I suggested in my comment, use GNU Parallel like this

parallel -k -L2 echo {1} {2} < file.txt

Output

line 1 line 2
line 3 line 4
line 5 line 6

where -k means "keep the output in order" and -L2 means "take 2 lines at a time from file.txt".

This has the advantage that, if you want to run 8 scripts at a time in parallel, you just specify -j 8 to parallel and the job is done.

回答2:

Well, it's not pretty and hopefully someone will offer a more elegant way to read successive pairs of lines in those loops, but one possibility is using a variable to track where you are, like

LOCATION=0
for file in first second last; do
    LOCATION=$((LOCATION+2))
    LINES=$(head -n $LOCATION $file | tail -n 2)
    # process lines

来源：https://stackoverflow.com/questions/29950596/parallel-processing-in-while-loop

标签

Linux

bash

shell

parallel-processing

ampersand