How can I have Java trigger C++ programs and vice versa when data is written to protocol buffers?

Question

Long story short, I have a Java process that reads and writes data to/from a process. I have a C++ program that takes the data, processes it and then needs to pass it back to Java so that Java can write it to a database.

The Java program pulls its data from Hadoop, so once the Hadoop process kicks off, it gets flooded with data but the actual processing(done by the C++ program) cannot handle all the data at once. So I need a way to control the flow as well. Also to complicate the problem(but simplify my work), I do the Java stuff and my friend does the C++ stuff and are trying to keep our programs as independent as possible.

That's the problem. I found Google protocol buffer and it seems pretty cool to pass data between the programs but I'm unsure how the Java Program saving data can trigger the c++ program to process and then when the c++ program saves the results how the Java program will be triggered to save the results (this is for one or a few records but we plan to process billions of records).

What is the best approach to this problem? Is there a simple way of doing this?

Answer 1

The simplest approach may be to use a TCP Socket connection. The Java program sends when you want to be done and the C++ program sends back the results.

Answer 2

Since you're going to want to scale this solution, i suggest using ZMQ.

Have your java app still pull the data from Hadoop.

It will then in turn push the data out using a PUSH socket.

Here you will have as many as needed c++ workers going who will process this data accepting connections as PULL sockets. This is scalable to as many different processor/cores/etc... that you need.

When each worker is finished it will push the results out on a PUSH socket to the 'storing' java program which is accepting info on a PULL socket.

It looks something like this example (standard divide and conquer methodology)

This process is scalable to as many workers as necessary as your first java program will block (but still is processing) when there aren't any available workers. So long as your ending java program is fast, you will see this scales really really nicely.

The emitting and saving program can be in the same program just use a zmq_poll device :)