Transferring data between executables

Question

I have two executables written in C++ on Windows. I generate some data in one, and want to call the other executable to process this data. I could write the data out to a file then read it in the other executable, but that seems rather expensive in terms of disk I/O. What is a better way of doing this? It seems like a simple enough question but google just isn't helping!

Let's say the data is around 100MB, and is generated in its entirety before needing to be sent (i.e. no streaming is needed).

Answers that work when mixing 32 bit and 64 bit processes gain bonus points.

Answer 1

If your processes can easily write to and read from file, just go ahead. Create the file with CreateFile and mark it as temporary & shareable. Windows uses this hint to delay physical writes, but all file semantics are still obeyed. Since your file is only 100 MB and actively in use, Windows is almost certainly able to keep its contents fully in RAM.

Answer 2

You can use Boost.MPI. It is from Boost, which has high quality standard, and the code sample seems pretty explicit:

http://www.boost.org/doc/libs/1_53_0/doc/html/mpi/tutorial.html#mpi.point_to_point

// The following program uses two MPI processes to write "Hello, world!"
// to the screen (hello_world.cpp):

int main(int argc, char* argv[])
{
  mpi::environment env(argc, argv);
  mpi::communicator world;

  if (world.rank() == 0) {
    world.send(1, 0, std::string("Hello"));
    std::string msg;
    world.recv(1, 1, msg);
    std::cout << msg << "!" << std::endl;
  } else {
    std::string msg;
    world.recv(0, 0, msg);
    std::cout << msg << ", ";
    std::cout.flush();
    world.send(0, 1, std::string("world"));
  }
  return 0;
}

Answer 3

Assuming you only want to go "one direction" (that is, you don't need to get data BACK from the child process), you could use _popen(). You write your data to the pipe and the child process reads the data from stdin.

If you need bidirectional flow of data, then you will need to use two pipes, one as input and one as output, and you will need to set up a scheme for how the child process connects to those pipes [you can still set up the stdin/stdout to be the data path, but you could also use a pair of named pipes].

A third option is a shared memory region. I've never done this in Windows, but the principle is pretty much the same as what I've used in Linux [and many years back in OS/2]: 1. Create a memory region with a given name in your parent process. 2. The child process opens the same memory region. 3. Data is stored by parent process and read by child process. 4. If necessary, semaphores or similar can be used to signal completion/results ready/etc.