Different MPI_Datatypes for Sender and Receiver

问题

With MPI, can data be sent and received with different MPI_Datatypes that are derived from the same base type and have the same total length?

Consider two MPI processes A and B. A has an array double a[n] and B has an array double b[m]. Both processes know that A wants to send B k doubles that are somehow distributed in a (only A has knowledge about this distribution). B (and only B) knows how it wants to arrange the k doubles in b. So both create (via MPI_Type_indexed and MPI_Type_commit) a datatype that for A corresponds to the elements it wants to send from a and for B corresponds to the locations it wants to save these elements to in b. A calls MPI_Isend(a, 1, A_datatype, ...) and B calls MPI_Irecv(b, 1, B_datatype, ...).

Does this work? If yes, is this the standard way to solve the problem at hand? If no, how is such a data exchange commonly done?

回答1:

Yes, it works, and it's quite a standard way to solve it.

The datatype is just instruction on how to pack data at the sender side, and instruction on how to unpack them at the receiver one.

This is a very useful and a very standard approach with MPI-IO.

回答2:

MPI types in matching send and receive operations have to be congruent. Congruency is a weaker form of equivalence. It means that the two datatypes might not be equal but their type signatures must match.

Each MPI datatype is a list of tuples in the form (offset[i], basic_type[i]) where offset[i] is the offset of the i-th item in bytes from the beginning of the data buffer (it could also be negative) and basic_type[i] is the language datatype of the item. The list of those tuples is called the type map. The list of basic types alone is called the signature of the type.

Imagine that you declare an indexed datatype with 5 blocks of one element each:

// Data type 1
MPI_Datatype type1;
int b_lens[5] = { 1, 1, 1, 1, 1 };
int b_offs[5] = { 1, 2, 3, 5, 7 };

MPI_Type_indexed(5, b_lens, b_offs, MPI_DOUBLE, &type1);

type1 has the following type map: { (8, double), (16, double), (24, double), (40, double), (56, double) } and a type signature of { double, double, double, double, double }.

Now a different contiguous type is created with the same number of basic elements:

// Data type 2
MPI_Datatype type2;

MPI_Type_contiguous(5, MPI_DOUBLE, &type2);

type2 has the following type map: { (0, double), (8, double), (16, double), (24, double), (32, double) }. This type map is obviously not the same as the one for type1 because the offsets are different. But the type signature is the same: { double, double, double, double, double }. Therefore type1 and type2 are said to be congruent.

The following piece of code is a perfectly valid MPI program:

double array1[10];
double array2[5];
double array3[10];

if (rank == rank_sender)
{
  MPI_Send(array1, 1, type1, rank_receiver, 0, MPI_COMM_WORLD);
  MPI_Send(array1, 1, type1, rank_receiver, 0, MPI_COMM_WORLD);
}
else if (rank == rank_receiver)
{
  MPI_Status status;
  int num_elems;

  MPI_Recv(array2, 1, type2, rank_sender, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
  MPI_Recv(array3, 10, MPI_DOUBLE, rank_sender, 0, MPI_COMM_WORLD, &status);
  MPI_Get_count(&status, MPI_DOUBLE, &num_elems); // num_elems will be set to 5
}

The MPI_Send call will pick the 1-th, 2-nd, 3-rd, 5-th, and 7-th elements of array1 at the sender and the MPI_Recv call will place them in the contiguous 5-elements array array2 at the receiver. Also a 10-element receive with the original datatype MPI_DOUBLE is congruent with the datatype of the sent message, though only the first 5 elements of the buffer will be filled in. A call to MPI_Get_count with a datatype of MPI_DOUBLE will then return 5.

Basically, congruency of both types means that the same number of data items of the same basic type(s) and in the same order are packed into and then unpacked from the message.

The MPI standard requires this congruency in correct applications. But in reality many MPI implementations do not transmit the type signature within the message and therefore the congruency is never enforced. For example, both Open MPI and MPICH allows you to use MPI_DOUBLE on the sender side and MPI_LONG (assuming an LP64 system) on the receiver side. The two datatypes are of the same size in bytes, but converting the double into long is meaningless. There are run-time MPI correctness checking tools like MUST that can detect such problems in the code.

来源：https://stackoverflow.com/questions/22329170/different-mpi-datatypes-for-sender-and-receiver

标签

mpi