如何在Spark Java Map Function中将sql.Date与SparkSQL一起使用

Question

I try to read a flat file (csv) that contains also Date values besides Strings and Integers. So instead of just using long/String for the Date fields would be nice to get an object that contains Date values.

The code that I have is the following:

JavaRDD<Date> dates = sc.textFile("hdfs://0.0.0.0:19000/Dates.csv").map(
    new Function<String, Date>(){
        @Override
        public Date call(String line){
            String[] fields = line.split(",");
            return Date.valueOf(fields[2]);
        }
});

DataFrame  schemaTransactions = sqlContext.createDataFrame(dates, Date.class);
schemaTransactions.registerTempTable("dates");
DataFrame dAs = sqlContext.sql("SELECT * FROM dates");
Row[] dARows = dAs.collect();

The code compiles but when it is executed, the error message

Exception in thread "main" java.lang.ClassCastException: org.apache.spark.sql.types.DateType$ cannot be cast to org.apache.spark.sql.types.StructType

is thrown which is confusing because the documentation sais java.sql.Date to be supported https://spark.apache.org/docs/latest/sql-programming-guide.html

The same error occurs when I use sql.Timestamp.

However, my initial goal was to use LocalDateTime from Java8, but since this is not supported I tried to use sql.Date.

Any suggestions or is it a bug?

Answer 1

Ok I just figuered out that if we place the Date object in a Wrapper class it seems to Work.

Here is the code: First we define our "Wrapper"

public class TestClass implements Serializable {

    Date date;

    public Date getDate() {
        return date;
    }

    public void setDate(Date date) {
        this.date = date;
    }

}

And then change the Type "Date" to the Wrapper class.

JavaRDD<TestClass> dates = sc.textFile("hdfs://0.0.0.0:19000/Dates.csv").map(
new Function<String, TestClass>(){
    @Override
    public TestClass call(String line){
        String[] fields = line.split(",");
        TestClass tc = new TestClass();
        tc.setDate(Date.parse(fields[2]));
        return tc;
    }
});

DataFrame  schemaTransactions = sqlContext.createDataFrame(dates, TestClass.class);
schemaTransactions.registerTempTable("dates");
DataFrame dAs = sqlContext.sql("SELECT * FROM dates");
dAs.count();

Maybe this is helpful for someone...