Spark-Java-在不使用Spark SQL数据框的情况下创建Parquet / Avro

Question

I want to get output of a Spark application(which we only use core Spark and people working on the project do not want to change it to Spark SQL) as Parquet or Avro files.

When I look for these two file types, I couldn't find any example without DataFrames, or in general Spark SQL. Can I achieve this without using SparkSQL?

My data is tabular, it has columns but in the processing, all data will be used, not a single column. It's columns are decided at runtime, so there is no "name,ID,adress" kinda generic columns. It looks like this:

No f1       f2       f3       ...
1, 123.456, 123.457, 123.458, ...
2, 123.789, 123.790, 123.791, ...
...

Answer 1

You can't save an rdd in parquet without converting it to dataframe. Rdd does not have schema but parquet file is in columnar format which needs schema, so we need to convert it to dataframe.

You can use createdataframe api

Answer 2

I tried this and it works like a champ...

public class ParquetHelper{

    static ParquetWriter<GenericData.Record> writer = null;
    private static Schema schema;

    public ParquetHelper(Schema schema, String pathName){

        try {
            Path path = new Path(pathName);
            writer = AvroParquetWriter.
                    <GenericData.Record>builder(path)
                    .withRowGroupSize(ParquetWriter.DEFAULT_BLOCK_SIZE)
                    .withPageSize(ParquetWriter.DEFAULT_PAGE_SIZE)
                    .withSchema(schema)
                    .withConf(new Configuration())
                    .withCompressionCodec(CompressionCodecName.SNAPPY)
                    .withValidation(true)
                    .withDictionaryEncoding(false)
                    .build();
            this.schema = schema;
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

     /*
     * 
     */
    public static void writeToParquet(JavaRDD<Record> empRDDRecords) throws IOException {

        empRDDRecords.foreach(record -> {
            if(null != record && new RecordValidator().validate(record, schema).isEmpty()){
                writeToParquet(record);
            }// TODO collect bad records here
        });

        writer.close();
    }

}