How to programmatically detect the number of cores and run an MPI program using all cores

Question

I do not want to use mpiexec -n 4 ./a.out to run my program on my core i7 processor (with 4 cores). Instead, I want to run ./a.out, have it detect

Answer 1

You can get number of CPUs by using for example this solution, and then start the MPI process by calling MPI_comm_spawn. But you will need to have a separate executable file.

Answer 2

This is quite a nontrivial thing to achieve in general. Also, there is hardly any portable solution that does not depend on some MPI implementation specifics. What follows is a sample solution that works with Open MPI and possibly with other general MPI implementations (MPICH, Intel MPI, etc.). It involves a second executable or a means for the original executable to directly call you library provided some special command-line argument. It goes like this.

Assume the original executable was started simply as ./a.out. When your library function is called, it calls MPI_Init(NULL, NULL), which initialises MPI. Since the executable was not started via mpiexec, it falls back to the so-called singleton MPI initialisation, i.e. it creates an MPI job that consists of a single process. To perform distributed computations, you have to start more MPI processes and that's where things get complicated in the general case.

MPI supports dynamic process management, in which one MPI job can start a second one and communicate with it using intercommunicators. This happens when the first job calls MPI_Comm_spawn or MPI_Comm_spawn_multiple. The first one is used to start simple MPI jobs that use the same executable for all MPI ranks while the second one can start jobs that mix different executables. Both need information as to where and how to launch the processes. This comes from the so-called MPI universe, which provides information not only about the started processes, but also about the available slots for dynamically started ones. The universe is constructed by mpiexec or by some other launcher mechanism that takes, e.g., a host file with list of nodes and number of slots on each node. In the absence of such information, some MPI implementations (Open MPI included) will simply start the executables on the same node as the original file. MPI_Comm_spawn[_multiple] has an MPI_Info argument that can be used to supply a list of key-value paris with implementation-specific information. Open MPI supports the add-hostfile key that can be used to specify a hostfile to be used when spawning the child job. This is useful for, e.g., allowing the user to specify via the GUI a list of hosts to use for the MPI computation. But let's concentrate on the case where no such information is provided and Open MPI simply runs the child job on the same host.

Assume the worker executable is called worker. Or that the original executable can serve as worker if called with some special command-line option, -worker for example. If you want to perform computation with N processes in total, you need to launch N-1 workers. This is simple:

(separate executable)

MPI_Comm child_comm;
MPI_Comm_spawn("./worker", MPI_ARGV_NULL, N-1, MPI_INFO_NULL, 0,
               MPI_COMM_SELF, &child_comm, MPI_ERRCODES_IGNORE);

(same executable, with an option)

MPI_Comm child_comm;
char *argv[] = { "-worker", NULL };
MPI_Comm_spawn("./a.out", argv, N-1, MPI_INFO_NULL, 0,
               MPI_COMM_SELF, &child_comm, MPI_ERRCODES_IGNORE);

If everything goes well, child_comm will be set to the handle of an intercommunicator that can be used to communicate with the new job. As intercommunicators are kind of tricky to use and the parent-child job division requires complex program logic, one could simply merge the two sides of the intercommunicator into a "big world" communicator that replaced MPI_COMM_WORLD. On the parent's side:

MPI_Comm bigworld;
MPI_Intercomm_merge(child_comm, 0, &bigworld);

On the child's side:

MPI_Comm parent_comm, bigworld;
MPI_Get_parent(&parent_comm);
MPI_Intercomm_merge(parent_comm, 1, &bigworld);

After the merge is complete, all processes can communicate using bigworld instead of MPI_COMM_WORLD. Note that child jobs do not share their MPI_COMM_WORLD with the parent job.

To put it all together, here is a complete functioning example with two separate program codes.

main.c

#include <stdio.h>
#include <mpi.h>

int main (void)
{
   MPI_Init(NULL, NULL);

   printf("[main] Spawning workers...\n");

   MPI_Comm child_comm;
   MPI_Comm_spawn("./worker", MPI_ARGV_NULL, 2, MPI_INFO_NULL, 0,
                  MPI_COMM_SELF, &child_comm, MPI_ERRCODES_IGNORE);

   MPI_Comm bigworld;
   MPI_Intercomm_merge(child_comm, 0, &bigworld);

   int size, rank;
   MPI_Comm_rank(bigworld, &rank);
   MPI_Comm_size(bigworld, &size);
   printf("[main] Big world created with %d ranks\n", size);

   // Perform some computation
   int data = 1, result;
   MPI_Bcast(&data, 1, MPI_INT, 0, bigworld);
   data *= (1 + rank);
   MPI_Reduce(&data, &result, 1, MPI_INT, MPI_SUM, 0, bigworld);
   printf("[main] Result = %d\n", result);

   MPI_Barrier(bigworld);

   MPI_Comm_free(&bigworld);
   MPI_Comm_free(&child_comm);

   MPI_Finalize();
   printf("[main] Shutting down\n");
   return 0;
}

worker.c

#include <stdio.h>
#include <mpi.h>

int main (void)
{
   MPI_Init(NULL, NULL);

   MPI_Comm parent_comm;
   MPI_Comm_get_parent(&parent_comm);

   int rank, size;
   MPI_Comm_rank(MPI_COMM_WORLD, &rank);
   MPI_Comm_size(MPI_COMM_WORLD, &size);
   printf("[worker] %d of %d here\n", rank, size);

   MPI_Comm bigworld;
   MPI_Intercomm_merge(parent_comm, 1, &bigworld);

   MPI_Comm_rank(bigworld, &rank);
   MPI_Comm_size(bigworld, &size);
   printf("[worker] %d of %d in big world\n", rank, size);

   // Perform some computation
   int data;
   MPI_Bcast(&data, 1, MPI_INT, 0, bigworld);
   data *= (1 + rank);
   MPI_Reduce(&data, NULL, 1, MPI_INT, MPI_SUM, 0, bigworld);

   printf("[worker] Done\n");
   MPI_Barrier(bigworld);

   MPI_Comm_free(&bigworld);
   MPI_Comm_free(&parent_comm);

   MPI_Finalize();
   return 0;
}

Here is how it works:

$ mpicc -o main main.c
$ mpicc -o worker worker.c
$ ./main
[main] Spawning workers...
[worker] 0 of 2 here
[worker] 1 of 2 here
[worker] 1 of 3 in big world
[worker] 2 of 3 in big world
[main] Big world created with 3 ranks
[worker] Done
[worker] Done
[main] Result = 6
[main] Shutting down

The child job has to use MPI_Comm_get_parent to obtain the intercommunicator to the parent job. When a process is not part of such a child job, the returned value will be MPI_COMM_NULL. This allows for an easy way to implement both the main program and the worker in the same executable. Here is a hybrid example:

#include <stdio.h>
#include <mpi.h>

MPI_Comm bigworld_comm = MPI_COMM_NULL;
MPI_Comm other_comm = MPI_COMM_NULL;

int parlib_init (const char *argv0, int n)
{
    MPI_Init(NULL, NULL);

    MPI_Comm_get_parent(&other_comm);
    if (other_comm == MPI_COMM_NULL)
    {
        printf("[main] Spawning workers...\n");
        MPI_Comm_spawn(argv0, MPI_ARGV_NULL, n-1, MPI_INFO_NULL, 0,
                       MPI_COMM_SELF, &other_comm, MPI_ERRCODES_IGNORE);
        MPI_Intercomm_merge(other_comm, 0, &bigworld_comm);
        return 0;
    }

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    printf("[worker] %d of %d here\n", rank, size);
    MPI_Intercomm_merge(other_comm, 1, &bigworld_comm);
    return 1;
}

int parlib_dowork (void)
{
    int data = 1, result = -1, size, rank;

    MPI_Comm_rank(bigworld_comm, &rank);
    MPI_Comm_size(bigworld_comm, &size);

    if (rank == 0)
    {
        printf("[main] Doing work with %d processes in total\n", size);
        data = 1;
    }

    MPI_Bcast(&data, 1, MPI_INT, 0, bigworld_comm);
    data *= (1 + rank);
    MPI_Reduce(&data, &result, 1, MPI_INT, MPI_SUM, 0, bigworld_comm);

    return result;
}

void parlib_finalize (void)
{
    MPI_Comm_free(&bigworld_comm);
    MPI_Comm_free(&other_comm);
    MPI_Finalize();
}

int main (int argc, char **argv)
{
    if (parlib_init(argv[0], 4))
    {
        // Worker process
        (void)parlib_dowork();
        printf("[worker] Done\n");
        parlib_finalize();
        return 0;
    }

    // Main process
    // Show GUI, save the world, etc.
    int result = parlib_dowork();
    printf("[main] Result = %d\n", result);
    parlib_finalize();

    printf("[main] Shutting down\n");
    return 0;
}

And here is an example output:

$ mpicc -o hybrid hybrid.c
$ ./hybrid
[main] Spawning workers...
[worker] 0 of 3 here
[worker] 2 of 3 here
[worker] 1 of 3 here
[main] Doing work with 4 processes in total
[worker] Done
[worker] Done
[main] Result = 10
[worker] Done
[main] Shutting down

Some things to keep in mind when designing such parallel libraries:

• MPI can only be initialised once. If necessary, call MPI_Initialized to check if the library has already been initialised.
• MPI can only be finalized once. Again, MPI_Finalized is your friend. It can be used in something like an atexit() handler to implement a universal MPI finalisation on program exit.
• When used in threaded contexts (usual when GUIs are involved), MPI must be initialised with support for threads. See MPI_Init_thread.