在Spark中将纯文本文件转换为Hadoop序列文件

Question

My existing project is using Hadoop map-reduce to generate a sequence file having a custom key and value which is in XML format.

The XML value is generated by reading one line at a time from the input source and the RecordReader is implemented to return the next value in XML format from the plain text.

eg Input source file has 3 rows (1st row is the header and rest rows is having actual data)

id|name|value
1|Vijay|1000
2|Gaurav|2000
3|Ashok|3000

Post the map method the sequence file has data as below:

FeedInstanceKey{feedInstanceId=1000, entity=bars}   <?xml version='1.0' encoding='UTF-8'?><bars><id>1</id><name>Vijay</name><value>1000</value></bars>
FeedInstanceKey{feedInstanceId=1000, entity=bars}   <?xml version='1.0' encoding='UTF-8'?><bars><id>2</id><name>Gaurav</name><value>2000</value></bars>
FeedInstanceKey{feedInstanceId=1000, entity=bars}   <?xml version='1.0' encoding='UTF-8'?><bars><id>3</id><name>Ashok</name><value>3000</value></bars>

Question: I wish to implement the same in Spark. Basically, read the input file and generate the key value pair as above.

Is there any way/possible to reuse the existing InputFormat and hence the RecordReader which is used in my Hadoop mapper class.

The RecordReader is responsible/having the logic to convert the plain text row to XML and return as value to Hadoop map method for writing in context.write() method.

Kindly suggest.

Answer 1

This is covered in the Spark documentation in the External Datasets section. The important part for you is:

For other Hadoop InputFormats, you can use the JavaSparkContext.hadoopRDD method, which takes an arbitrary JobConf and input format class, key class and value class. Set these the same way you would for a Hadoop job with your input source. You can also use JavaSparkContext.newAPIHadoopRDD for InputFormats based on the “new” MapReduce API (org.apache.hadoop.mapreduce).

Here's a simple example demostrating how to use it:

public final class ExampleSpark {

    public static void main(String[] args) throws Exception {
        JavaSparkContext spark = new JavaSparkContext();
        Configuration jobConf = new Configuration();

        JavaPairRDD<LongWritable, Text> inputRDD = spark.newAPIHadoopFile(args[0], TextInputFormat.class, LongWritable.class, Text.class, jobConf);
        System.out.println(inputRDD.count());

        spark.stop();
        System.exit(0);
    }
}

You can see the Javadocs for JavaSparkContext here .