MPI BCast (broadcast) of a std::vector of structs

Question

I've got a question regarding passing of a std::vector of structs via MPI.

First off, details. I'm using OpenMPI 1.4.3 (MPI-2 compliant) with gcc. Note that I can't use boost MPI or OOMPI -- I'm bound to using this version.

I've got a struct to aggregate some data:

  struct Delta {
    Delta() : dX(0.0), dY(0.0), dZ(0.0) {};
    Delta(double dx, double dy, double dz) :
      dX(dx), dY(dy), dZ(dz) {};
    Delta(const Delta& rhs) :
      dX(rhs.dX), dY(rhs.dY), dZ(rhs.dZ) {};

    double dX;
    double dY;
    double dZ;
  };

  typedef std::vector<Delta> DeltaLine;

and I have a DeltaLine that I'd like to broadcast, via MPI, to all the nodes.

Can I do the following safely and portably? This works for me in my test case. I just want to make sure it's legal and kosher across different platforms and according to the C++ and MPI standards.

Thanks! Madeleine.

  //Create an MPI struct for the Delta class
  const int    nItems=3;
  int          blocklengths[nItems] = {1, 1, 1};
  MPI_Datatype types[nItems] = {MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE};
  MPI_Datatype MPI_DeltaType;
  MPI_Aint     offsets[nItems];

  offsets[0] = offsetof(Delta, dX);
  offsets[1] = offsetof(Delta, dY);
  offsets[2] = offsetof(Delta, dZ);

  MPI_Type_create_struct(nItems, blocklengths, offsets, types, &MPI_DeltaType);
  MPI_Type_commit(&MPI_DeltaType);

  //This is the vector to be filled, and its size
  DeltaLine deltaLine;
  unsigned deltaLineSize;

  //If this is the master proc, get the DeltaLine and its size
  if(amMaster()) {
    deltaLine = getMasterDeltaLine();
    deltaLineSize = deltaLine.size();
  }

  //Send out the correct size
  MPI_Bcast(&deltaLineSize, 1, MPI_UNSIGNED, COMM_PROC, MPI_COMM_WORLD);

  //Size the delta line vector, and broadcast its contents
  deltaLine.reserve(deltaLineSize);
  MPI_Bcast(&deltaLine.front(), deltaLineSize, MPI_DeltaType, COMM_PROC, MPI_COMM_WORLD);

  //Free up the type
  MPI_Type_free(&MPI_DeltaType);

Answer 1

The C++ standard guarantees that the elements of std::vector are stored contiguously in memory and that std::vector::reserve() (re-)allocates memory if necessary at the time of call, therefore your solution is perfectly valid from memory management point of view. Though, as Solkar noted, std::vector::reserve() only reserves memory space but the vector object is not aware that there is data being directly written in that memory and therefore keeps the previous element count (zero for freshly created vectors). This can be fixed by calling std::vector::resize() before the second broadcast operation.

One comment though that applies to all cases when constructed MPI datatypes are used to send arrays - you should take care of possible padding between the consecutive array elements. In other words, it is possible for the following to hold because of possible padding at the end of the struct:

(char*)&deltaLine[1] - (char*)&deltaLine[0] != mpi_extentof(MPI_DeltaType)

where mpi_extentof is the extent of the MPI datatype as returned by MPI_Type_get_extent(). Because MPI uses the extent to determine where each array element starts, it is advisable to explicitly set it for any structure type that is used to send more than one element. With MPI-1 this is typically done by adding one special structure element of the MPI_UB pseudotype, but in modern MPI code (or in MPI-2 in general) one should use MPI_Type_create_resized for that purpose:

//Create an MPI struct for the Delta class
const int    nItems=3;
int          blocklengths[nItems] = {1, 1, 1};
MPI_Datatype types[nItems] = {MPI_DOUBLE, MPI_DOUBLE, MPI_DOUBLE};
MPI_Datatype MPI_DeltaType_proto, MPI_DeltaType;
MPI_Aint     offsets[nItems];

offsets[0] = offsetof(Delta, dX);
offsets[1] = offsetof(Delta, dY);
offsets[2] = offsetof(Delta, dZ);

MPI_Type_create_struct(nItems, blocklengths, offsets, types, &MPI_DeltaType_proto);

// Resize the type so that its length matches the actual structure length

// Get the constructed type lower bound and extent
MPI_Aint lb, extent;
MPI_Type_get_extent(MPI_DeltaType_proto, &lb, &extent);

// Get the actual distance between to vector elements
// (this might not be the best way to do it - if so, substitute a better one)
extent = (char*)&deltaLine[1] - (char*)&deltaLine[0];

// Create a resized type whose extent matches the actual distance
MPI_Type_create_resized(MPI_DeltaType_proto, lb, extent, &MPI_DeltaType);
MPI_Type_commit(&MPI_DeltaType);

In your case there are only double elements in the structure and no padding is expected, therefore doing this all is not necessary. But keep it in mind for your future work with MPI.

Answer 2

std::vector::reserve(N) does not affect size but (if at all) capacity (and maybe location), so for the receiving containers deltaLine will still be a zero size vector, regardless of its capacity equaling deltaLineSize.

That's not yet a problem in the code as-is, but I assume you intend to do some processing using the received data.

I would also check the return value of (at least) the first MPI_BCast, because if that, for whatever reason fails on one process, the vector's size there would be 0, and if it responds to the second broadcast bounds will be violated.