How to use mpirun to use different CPU cores for different programs?

Question

I have a virtual machine with 32 cores. I am running some simulations for which I need to utilize 16 cores at one time.

I use the below command to run a job on 16 cores :

mpirun -n 16 program_name args > log.out 2>&1

This program runs on 16 cores.

Now if I want to run the same programs on the rest of the cores, with different arguments, I use the same command like

mpirun -n 8 program_name diff_args > log_1.out 2>&1

The second process utilizes the same 16 cores that were utilized earlier. How can use mpirun to run this process on 8 different cores, not the previous 16 that first job was using.

I am using headless Ubuntu 16.04.

Answer 1

Open MPI's launcher supports restricting the CPU set via the --cpu-set option. It accepts a set of logical CPUs expressed as a list of the form s0,s1,s2,..., where each list entry is either a single logical CPU number of a range of CPUs n-m.

Provided that the logical CPUs in your VM are numbered consecutively, what you have to do is:

mpirun --cpu-set  0-15 --bind-to core -n 16 program_name args > log.out 2>&1
mpirun --cpu-set 16-23 --bind-to core -n  8 program_name diff_args > log_1.out 2>&1

--bind-to core tells Open MPI to bind the processes to separate cores each while respecting the CPU set provided in the --cpu-set argument.

It might be helpful to use a tool such as lstopo (part of the hwloc library of Open MPI) to obtain the topology of the system, which helps in choosing the right CPU numbers and, e.g., prevents binding to hyperthreads, although this is less meaningful in a virtualised environment.

(Note that lstopo uses a confusing naming convention and calls the OS logical CPUs physical, so look for the numbers in the (P#n) entries. lstopo -p hides the hwloc logical numbers and prevents confusion.)