通过Java中的套接字发送（并行）流

Question

I'm trying to write a distributed Map-reduce program in java, using the Java 8 map-reduce framework, that has the following design:

One client sends data to 3 Mappers (each a different machine/standalone java application). The mappers work with the data in parallel by creating a parallelStream() from the List of data.

Now each mapper should call .map(...) on its parallelStream. Then, the idea is to send the mapped data to another node, the Reducer.

The reducer will get the Stream and call .reduce(...) on it and finally .get() to get the final results, that are sent back to the client.

My program works if I call .map(...).reduce(...).get() on the same program, but I want to be able to have a separate reducer node.

As I am new in socket programming and also at using Java 8, I'm having trouble sending the stream through the Socket, because it throws a "java.io.NotSerializableException: java.util.stream.ReferencePipeline$3" the moment I try to write the stream with WriteObject .

What's the best way to proceed here? Can I turn the stream into something else, send it and then turn it into a stream again on my Reducer node? Is there a better way to send the stream than through an ObjectOutputStream ?

Any ideas are very much appreciated. Thank you very much!

PS: The stream is a Stream<Map<String, Integer>> .

Answer 1

One approach is to terminate the map node with a forEach that pushes data into the socket. This strategy is superior to a collection approach if the collection could be very large (or theoretically infinite); it's space efficient, it's buffered, and down stream nodes are not idle waiting for collection process to complete.

Next wrap the socket reader for the reduce node in a Spliterator (extend AbstractSpliterator). The tryAdvance method of Spliterator reads data from the socket and makes it available to the stream through a caller provided Consumer. tryAdvance returns false when there is no more data (your end-of-stream marker, socket end of stream, or socket exception). AbstractSpliterator.trySplit implements limited parallelism.

Use StreamSupport.stream(Spliterator spliterator, boolean parallel) to construct a stream from your Spliterator implementation. Your reduce operation pulls its data from this stream.

You could retain the socket and your end-of-stream marker could be more like an end-of-message marker (reminds me of a batching pig in a pipeline).