具有二进制输入的Hadoop流作业？

Question

I wish to convert a binary file in one format to a SequenceFile.

I have a Python script that takes that format on stdin and can output whatever I want.

The input format is not line-based. The individual records are binary themselves, hence the output format cannot be \\t delimited or broken into lines with \\n.

Can I use the Hadoop Streaming interface to consume a binary format? How do I produce a binary output format?

I assume the answer is "No" unless I hear otherwise.

Answer 1

You may consider using NullWritable as output, and generating the SequenceFile directly inside of your python script. You can look up the hadoop-python project in github to see candidate code: though it is admittedly bit large-ish/heavy it does handle the sequencefile generation.