SLURM sbatch job array for the same script but with different input arguments run in parallel

Question

I have a problem where I need to launch the same script but with different input arguments.

Say I have a script myscript.py -p -i

Answer 1

According to this page, job arrays incur significant overhead:

If the running time of your program is small, say ten minutes or less, creating a job array will incur a lot of overhead and you should consider packing your jobs.

That page provides a few examples to run your kind of job, using both arrays and "packed jobs."

If you don't want/need to specify the resources for your job, here is another approach: I'm not sure if it's a usecase that was intended by Slurm, but it appears to work, and the submission script looks a little bit nicer since we don't have to linearize the indices to fit it into the job-array paradigm. Plus it works well with nested loops of arbitrary depth.

Run this directly as a shell script:

#!/bin/bash
FLAGS="--ntasks=1 --cpus-per-task=1"
for i in 1 2 3 4 5; do
        for j in 1 2 3 4 5; do
            for k in 1 2 3 4 5; do
                sbatch $FLAGS testscript.py $i $j $k
        done
    done
done

where you need to make sure testscript.py points to the correct interpreter in the first line using the #! e.g.

#!/usr/bin/env python 
import time
import sys
time.sleep(5)
print "This is my script"
print sys.argv[1], sys.argv[2], sys.argv[3] 

Alternatively (untested), you can use the --wrap flag like this

sbatch $FLAGS --wrap="python testscript.py $i $j $k"

and you won't need the #!/usr/bin/env python line in testscript.py