在Windows上从Java到C ++的高效数据传输

Question

I'm looking to stream lots of data (up to ~1 Gbit) from Java to a C++ application (both on the same machine). I'm currently using a FIFO on Linux but need a Windows solution too.

The most cross-platform method seems to be a local socket, but: a) won't I get huge overhead from TCP checksumming and copying to & from kernel space, and b) won't the average user's firewall try to inspect the and maybe block the connection?

It seems like a safer solution may be to use JNI and the Named Pipe API (\\.\\pipe\\blah), making a god-awful platform-specific mess of both sides of the connection.

Are these really my 2 best options (and which would people recommend?) Thanks!

Answer 1

您应该看一下Google的Protocol Buffers ，它支持C ++和Java。

Answer 2

Named pipes would be more efficient than TCP, but how about just using shared memory blocks?

I don't know what primitives exist on the Java side for interfacing with shared memory, but from the C++ side it would be more efficient to access data in shared memory than read it out of either a socket or named pipe. You would have to implement your own flow control and blocking primitives, but these could be fairly straight forward.

Answer 3

I would use a local socket, which is, as you state, the most cross-platform method.

Kernel-User space copies should not be an issue since any other method you could choose would requiere this kind of copy except for maybe shared memory. It is available on every Unix system and also Windows has its way of doing it

To use shared memory in Java the only way is to implement it by means of your own .DLL/.SO and JNI to access it.

Answer 4

Your fastest solution will be memory mapping a shared segment of memory, and them implementing a ring-buffer or other message passing mechanism. In C++ this is straight forward, and in Java you have the FileChannel.ma p method which makes it possible.

The next alternative would be to use the stdin/stdout of the two processes. If one can exec the other, this can be quite fast.

Lastly, as you've noted, you can do socket IO. For streaming video this isn't a great option, but if your passing XML around, the overhead will be minimal in comparison to the other processing.

Answer 5

If you're happy with writing JNI, consider Boost.Interprocess . This will give you portable shared memory on both Linux and Windows. Remember that there's no kernel roundtrip for reading/writing shared memory.

Answer 6

If it's a big chunk of data in "one" function call I would recommend JNI.

Take a look at this: Sharing output streams through a jni interface

Snippet from the article, it transfert data from c++ to java, the opposite would be also easy to do:

In all, the general strategy for sharing binary data (A/V files, images, etc.) from C with Java requires byte arrays. You create a Java byte array in C like this:

const char[] rawData = {0,1,2,3,4,5,6,7,8,9}; //Or get some raw data from somewhere
int dataSize = sizeof(rawData);
printf("Building raw data array copy\n");
jbyteArray rawDataCopy = env->NewByteArray(dataSize);
env->SetByteArrayRegion(rawDataCopy, 0, dataSize, rawData);

And pass it to Java like this:

printf("Finding callback method\n");
//Assumes obj is the Java instance that will receive the raw data via callback
jmethodID aMethodId = env->GetMethodID(env->GetObjectClass(obj),"handleData","([B)V");
if(0==aMethodId) throw MyRuntimeException("Method not found error");
printf("Invoking the callback\n");
env->CallVoidMethod(obj,aMethodId, &rawDataCopy);

you would have a Java object that looked something like this:

public class MyDataHandler {
  OutputStream dataStream;
  public MyDataHandler(OutputStream writeTo) { dataStream = writeTo;}
  public void handleData(byte[] incomingData) { dataStream.write(incomingData); }
}

That handler would be passed to C via native method like so:

public class NativeIntegration {
  public native void generateBinaryWithHandler(MyDataHandler handler);

  //Here we assume response is something like a network stream
  public void doCallNativeFunction(ResponseStream response) {
    MyDataHandler handler = new MyDataHandler(response);
    generateBinaryWithHandler(handler);
  }
}

Also, you can use other technologies: CORBA, ASN.1 ( ASN.1 tool ), UDP or TCP

Answer 7

I would use a local socket with negative acknowledgment UDP if the bit rate is going to be too high for TCP (although I would try TCP first and confirm that it is a problem). There should be minimal, if any, packets dropped if you are doing the streaming on the same machine, but the addition of a negative acknowledgement layer will take care of that case for you.

Answer 8

How about using System.out and System.in?

If that isn't suitable, then Sockets is your best bet.

Answer 9

If your C++ process starts the Java process, it might benefit from the inheritedChannel . Also, if the Java process is using a file, I recommend exploration of the transferTo and transferFrom methods. When doing file to file IO, these avoid needlessly tripping back and forth between user and kernel space; the same optimizations might kick in if you're using a special socket channel.

Answer 10

我建议使用UDP“连接”来确认每个没有故障的第N个数据包，并请求重传它将丢失的少数数据包。

Answer 11

I would advise against JNI, because it is very difficult to debug. If the C++ code segfaults or throws an uncaught exception, your JVM will crash, and you will have no idea why.