obtain result of map-reduce job as stream

Question

Iam writing a map-reduce job in Java I would like to know is it possible to obtain output of the job as stream(may be a output stream) rather a physical output file. My objective is to use the stream for another application.

Answer 1

You can write a Custom Output Format and use that write to any stream you want to. Not necessarily a file. See this tutorial on how to write a Custom Output Format.

Or else you can make use Hadoop Streaming API. Have a look here for that.

Answer 2

I don't think you can do this using Apache-Hadoop. It is designed to work in a distributed system and AFAIK providing the way to emit an output stream would defy the purpose, as then how system would decide on the stream to emit, ie of which reducer! You may write to a flat-file/DB/amazon-s3 etc but perhaps you won't get a stream .