Efficient data transfer from Java to C++ on windows

Question

I\'m looking to stream lots of data (up to ~1 Gbit) from Java to a C++ application (both on the same machine). I\'m currently using a FIFO on Linux but need a Windows soluti

Answer 1

I recommend a UDP "connection" that acknowledges every Nth packet that was received without fault, and requests a retransmission of the few packets it will miss.

Answer 2

I would advise against JNI, because it is very difficult to debug. If the C++ code segfaults or throws an uncaught exception, your JVM will crash, and you will have no idea why.

Answer 3

If it's a big chunk of data in "one" function call I would recommend JNI.

Take a look at this:Sharing output streams through a jni interface

Snippet from the article, it transfert data from c++ to java, the opposite would be also easy to do:

In all, the general strategy for sharing binary data (A/V files, images, etc.) from C with Java requires byte arrays. You create a Java byte array in C like this:

const char[] rawData = {0,1,2,3,4,5,6,7,8,9}; //Or get some raw data from somewhere
int dataSize = sizeof(rawData);
printf("Building raw data array copy\n");
jbyteArray rawDataCopy = env->NewByteArray(dataSize);
env->SetByteArrayRegion(rawDataCopy, 0, dataSize, rawData);

And pass it to Java like this:

printf("Finding callback method\n");
//Assumes obj is the Java instance that will receive the raw data via callback
jmethodID aMethodId = env->GetMethodID(env->GetObjectClass(obj),"handleData","([B)V");
if(0==aMethodId) throw MyRuntimeException("Method not found error");
printf("Invoking the callback\n");
env->CallVoidMethod(obj,aMethodId, &rawDataCopy);

you would have a Java object that looked something like this:

public class MyDataHandler {
  OutputStream dataStream;
  public MyDataHandler(OutputStream writeTo) { dataStream = writeTo;}
  public void handleData(byte[] incomingData) { dataStream.write(incomingData); }
}

That handler would be passed to C via native method like so:

public class NativeIntegration {
  public native void generateBinaryWithHandler(MyDataHandler handler);

  //Here we assume response is something like a network stream
  public void doCallNativeFunction(ResponseStream response) {
    MyDataHandler handler = new MyDataHandler(response);
    generateBinaryWithHandler(handler);
  }
}

Also, you can use other technologies: CORBA, ASN.1 (ASN.1 tool), UDP or TCP

Answer 4

If your C++ process starts the Java process, it might benefit from the inheritedChannel. Also, if the Java process is using a file, I recommend exploration of the transferTo and transferFrom methods. When doing file to file IO, these avoid needlessly tripping back and forth between user and kernel space; the same optimizations might kick in if you're using a special socket channel.

Answer 5

If you're happy with writing JNI, consider Boost.Interprocess. This will give you portable shared memory on both Linux and Windows. Remember that there's no kernel roundtrip for reading/writing shared memory.

Answer 6

I would use a local socket, which is, as you state, the most cross-platform method.

Kernel-User space copies should not be an issue since any other method you could choose would requiere this kind of copy except for maybe shared memory. It is available on every Unix system and also Windows has its way of doing it

To use shared memory in Java the only way is to implement it by means of your own .DLL/.SO and JNI to access it.