bash: how to keep some delay between multiple instances of a script

问题

I am trying to download 100 files using a script

I dont want at any point of time not more than 4 downloads are happening.

So i have create a folder /home/user/file_limit. In the script it creates a file here before the download and after the download is complete it will delete it.

The script will check the number of files in the folder is less than 4 then only it will allow to create a file in the folder /home/user/file_limit

I am running a script like this

    today=`date +%Y-%m-%d-%H_%M_%S_%N`;
    while true
    do
        sleep 1
        # The below command will find number of files in the folder /home/user/file_limit
        lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l)
        if [ $lines -lt 5 ]; then
            echo "Create file"
            touch "/home/user/file_limit/download_${today}"
            break;
        else
            echo "Number of files equals 4"
        fi
    done

    #After this some downloading happens and once the downloading is complete

    rm "/home/user/file_limit/download_${today}"

The problem i am facing is when 100 such scripts are running. Eg when the number of files in the folder are less than 4, then many touch "/home/user/file_limit/download_${today}" gets executed simultaneously and all of them creates files. So the total number of files become more than 4 which i dont want because more downloads cause my system get slower.

How to ensure there is a delay between each script for checking the lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l) so that only one touch command get executed.

Or HOw to ensure the lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l) command is checked by each script in a queue. No two scripts can check it at the same time.

回答1:

How to ensure there is a delay between each script for checking the lines=$(find ... | wc -l) so that only one touch command get executed

Adding a delay won't solve the problem. You need a lock, mutex, or semaphore to ensure that the check and creation of files is executed atomically.

Locks limit the number of parallel processes to 1. Locks can be created with flock (usually pre-installed). Semaphores are generalized locks limiting the number concurrent processes to any number N. Semaphores can be created with sem (part of GNU parallel, has to be installed).

The following script allows 4 downloads in parallel. If 4 downloads are running and you start the script a 5th time then that 5th download will pause until one of the 4 running downloads finish.

#! /usr/bin/env bash

main() {
  # put your code for downloading here
}
export -f main
sem --id downloadlimit -j4 main

回答2:

My solution starts maximum MAXPARALELLJOBS number of process and waits until all of those processes are done...

Hope it helps your problem.

MAXPARALELLJOBS=4
count=0
while <not done the job>
 do
 ((count++))
 ( <download job> ) &
 [ ${count} -ge ${MAXPARALELLJOBS} ] && count=0 && wait
 done

来源：https://stackoverflow.com/questions/61767478/bash-how-to-keep-some-delay-between-multiple-instances-of-a-script

标签

Linux

bash

queue