Can't find process listed by `jobs` command

问题

I am using jobs command to control the number of compute-intensive processes. I want to not run more than max_cnt processes at a time, and stop only when all the processes have stopped.

I use below bash script to accomplish this. However, this code always lists one process as running even after everything has executed and stopped.

Moreover, I can't find that process listed in htop's list of processes. What should I do or where should I look for that process that is listed by the result of echo $(jobs -p) command and how should I fix this issue of not exiting even when everything stops.

#!/usr/bash

SLEEP=5
max_cnt=8

# generate a random number less than eq $1
function random {
    rand=$RANDOM
    while [ "$rand" -gt $1 ]
    do
        rand=$RANDOM
    done
}

function job {
    # resource intensive job simulated by random sleeping
    echo param1="$1",param2="$2"
    random 20
    echo Sleeping for $rand
    sleep $rand
}

for param1 in 1e-6 1e-5 1e-4 1e-3 1e-2
do
    for param2 in "ones" "random"
    do
        echo starting job with $param1 $param2
        job $param1 $param2 &
        while [ "$(jobs -p|wc -l)" -ge "$max_cnt" ]
        do
            echo "current running jobs.. $(jobs -p|wc -l) ... sleeping"
            sleep $SLEEP
        done
    done
done

while [ "$(jobs -p|wc -l)" -ge 1 ]
do
    echo "current running jobs.. $(jobs -p|wc -l) ... sleeping"
    sleep $SLEEP
    echo $(jobs -p)
done

回答1:

As mentioned in the comments, you may want to consider using GNU Parallel, it makes life easier when managing parallel jobs. Your code could look like this:

#!/usr/bin/env bash

function job {
    # resource intensive job simulated by random sleeping
    echo param1="$1",param2="$2"
    ((s=(RANDOM%5)+1))
    echo Sleeping for $s
    sleep $s
}
# export function to subshells
export -f job

parallel -j8 job {1} {2} ::: 1e-6 1e-5 1e-4 1e-3 1e-2 ::: "ones" "random"

Sample Output

param1=1e-6,param2=ones
Sleeping for 1
param1=1e-5,param2=ones
Sleeping for 1
param1=1e-2,param2=ones
Sleeping for 1
param1=1e-4,param2=ones
Sleeping for 2
param1=1e-4,param2=random
Sleeping for 2
param1=1e-6,param2=random
Sleeping for 4
param1=1e-2,param2=random
Sleeping for 3
param1=1e-3,param2=random
Sleeping for 4
param1=1e-5,param2=random
Sleeping for 5
param1=1e-3,param2=ones
Sleeping for 5

There are many other switches and parameters:

parallel --dry-run ... will show you what it would do, without actually doing anything
parallel --eta ... which gives you an "Estimated Time of Arrival"
parallel --bar ... which gives you a progress bar
parallel -k ... which keeps output in order
parallel -j 8 ... which runs 8 jobs at a time rather than the default of 1 job per CPU core
parallel --pipepart ... which will split the contents of a massive file across subprocesses

Note also that GNU Parallel can distribute work across other machines in your network, and it has fail and retry handling, output tagging and so on...

来源：https://stackoverflow.com/questions/60311034/cant-find-process-listed-by-jobs-command

标签

Linux

bash

jobs