SLURM sbatch job array for the same script but with different input string arguments run in parallel

问题

My question is similar with this one, and the difference is that my different arguments are not numbers but strings.

If I have a script (myscript.R) that takes two strings as arguments: "text-a", "text-A". My shell script for sbatch would be:

#!/bin/bash

#SBATCH -n 1
#SBATCH -c 12
#SBATCH -t 120:00:00
#SBATCH --partition=main
#SBATCH --export=ALL

srun ./myscript.R "text-a" "text-A"

Now I have a few different input strings that I'd like to run with:

first <- c("text-a","text-b","text-c","text-d")
second <- c("text-A","text-B","text-C","text-D")

and I want to run myscript.R with combinations of the texts, for example:

srun ./myscript.R "text-a" "text-A"
srun ./myscript.R "text-b" "text-B"
srun ./myscript.R "text-c" "text-C"
srun ./myscript.R "text-d" "text-D"

But if I put them in the same shell script, they'll run sequentially. I only know that I can use #SBATCH -a 0-10 when the arguments are index. If I want to submit the four scripts at the same time and each of them with the exact same settings (especially each one need to be assigned -c 12), how can I do that?

Thanks!

回答1:

You can store de list of argument values in an array and use the SLURM_ARRAY_TASK_ID env variable to index that array.

#!/bin/bash

#SBATCH -n 1
#SBATCH -c 12
#SBATCH -t 120:00:00
#SBATCH --partition=main
#SBATCH --export=ALL
#SBATCH --array=0-3

A=(text-{a..d}) # This is equivalent to A=(text-a text-b ... text-d)
B=(text-{A..D})

srun ./myscript.R "${A[$SLURM_ARRAY_TASK_ID]}" "${B[$SLURM_ARRAY_TASK_ID]}"

and simply submit it with sbatch.

来源：https://stackoverflow.com/questions/43642029/slurm-sbatch-job-array-for-the-same-script-but-with-different-input-string-argum

标签

arrays

slurm

sbatch