In Perl 5, how to queue a process/application after it reaches a maximum limit

Question

I have a program that is being symlinked to multiple directories e.g.

/main/foo.pl
/run1/foo.pl -> /main/foo.pl
/run2/foo.pl -> /main/foo.pl
/run3/foo.pl -> /main/foo.pl
/run4/foo.pl -> /main/foo.pl

They're being run as cron jobs, hence I have the following entries in the crontab:

*/2 * * * * /run1/foo.pl
*/2 * * * * /run2/foo.pl
*/2 * * * * /run3/foo.pl
*/2 * * * * /run4/foo.pl

A snippet of foo.pl is as below:

use Fcntl qw(:flock);
use autodie qw(:all);
open my $self, '>', "$FindBin::Bin/lockme";
flock( $self, LOCK_EX|LOCK_NB )
    or die "Cannot acquire lock, already running!";

{
    my $long_proc = Process->new();
    $long_proc->run();
}        

You get the idea, each of the cron process can only run once because there is a lock semaphore check. But run1, run2, run3, and run4 can run simultaneously.

Now what I need is that I want to limit the number of process to maximum of four. If someone adds another cron processes like:

New symlinks:

/run5/foo.pl -> /main/foo.pl
/run6/foo.pl -> /main/foo.pl

Additional crontab:

*/5 * * * * /run5/foo.pl
* * * * * /run6/foo.pl

Both run5 and run6 need to be queued whenever run1, run2, run3, and run4 are all still running. Thus at any given time there will be only 4 processes run.

How can I achieve that? Is there any CPAN module that handles it?

Thanks!

Answer 1

You have a few general approaches:

Quick and Dirty

Have a directory with N lockfiles, and have each process try to lock each file in turn, sleeping a bit.

Caveat: It's a busy loop, yes, but it might be good enough.

Daemonic Redesign

Turn foo.pl into a local socket-listening daemon, as it is easy for a parent process to manage concurrency of its children. Turn your cron jobs into blocking requests for the daemon to so something. (See, e.g., Parallel::ForkManager).

Caveat: You'll have to carefully re-implement setuid/privilege changing code if you were using cron to run foo.pl as multiple different users.

Interprocess Semaphores, SysV

IPC::Semaphore is a somewhat more convenient interface to the SysV semget family of functions in perlfunc, and the SEM_UNDO flag means that the unexpected process death of one of your workers won't "waste" a semaphore increment.

Caveat: You'll likely need to create and initialize the SysV semaphore in a separate process, distinct from foo.pl, since you cannot create and initialize a SysV semaphore in an atomic operation.

Interprocess Semaphores, POSIXly

POSIX::RT::Semaphore also supports interprocess semaphores. The API is simpler than SysV, and semaphores can be exclusively created and initialized to a value in an atomic syscall.

This is probably what I'd use.

Answer 2

Please see comment from @tsee in @pilcrow answer above. Steffen Muller's IPC::ConcurrencyLimit can control the number of concurrenct processes on cooperative multi processing nicely.