如何使用python读取hadoop映射文件？

Question

I have map file that is block compressed using DefaultCodec. The map file is created by java application like this:

MapFile.Writer writer =
            new MapFile.Writer(conf, path,
                    MapFile.Writer.keyClass(IntWritable.class),
                    MapFile.Writer.valueClass(BytesWritable.class),
                    MapFile.Writer.compression(SequenceFile.CompressionType.BLOCK, new DefaultCodec()));

This file is stored in hdfs and I need to read some key,values from it in another application using python. I can't find any library that can do that. Do you have any suggestion and example?

Thanks

Answer 1

I would suggest using Spark which has a function called textFile() which can read files from HDFS and turn them into RDDs for further processing using other Spark libraries.

Here's the documentation : Pyspark

Answer 2

Create a reader as follow:

path = '/hdfs/path/to/file'
key = LongWritable()
value = LongWritable()
reader = MapFile.Reader(path)
while reader.next(key, value):
        print key, value

Check out these hadoop.io.MapFile Python examples

And available methods in MapFile.py