将RDD [List [AnyRef]]转换为RDD [List [String，Date，String，String]]

Question

I want to set return type of RDD. But it is RDD[List[AnyRef]]. So I am not able to specify anything directly. Like,

val rdd2 = rdd1.filter(! _.isEmpty).filter(x => x(0) != null)

This returns RDD[List[String, Date, String, String]] type of RDD but it was RDD[List[AnyRef]].

EDIT

rdd1:
List(Sun Jul 31 10:21:53 PDT 2016, pm1, 11, ri1)
List(Mon Aug 01 12:57:09 PDT 2016, pm3, 5, ri1)
List(Mon Aug 01 01:11:16 PDT 2016, pm1, 1, ri2)

This rdd1 is RDD[List[AnyRef]] type.

Now I want rdd2 in this type:

RDD[List[Date, String, Long, String]]

The reason is that I am facing issues with date while converting RDD to Data Frame using schema. To deal with that firstly I have to fix the RDD type. That problem's solution is : Spark rdd correct date format in scala?

Answer 1

Here is a small example which leads to the same problem (I omitted Date , replaced it by String , that's not the point):

val myRdd = sc.makeRDD(List(
  List[AnyRef]("date 1", "blah2", (11: java.lang.Integer), "baz1"),
  List[AnyRef]("date 2", "blah3", (5: java.lang.Integer),  "baz2"),
  List[AnyRef]("date 3", "blah4", (1: java.lang.Integer),  "baz3") 
))

// myRdd: org.apache.spark.rdd.RDD[List[AnyRef]] = ParallelCollectionRDD[0]

Here is how you can recover the types:

val unmessTypes = myRdd.map{
  case List(a: String, b: String, c: java.lang.Integer, d: String) => (a, b, (c: Int), d)
}

// unmessTypes: org.apache.spark.rdd.RDD[(String, String, Int, String)] = MapPartitionsRDD[1]

You simply apply a partial function that matches lists of length 4 with elements of specified types, and constructs the tuples of expected type out of it. If your RDD indeed contains only lists of length 4 with the expected types, the partial function will never fail.

Answer 2

By looking at your Spark rdd correct date format in scala? , it seems that you are having issue in converting your rdd to dataframe. Tzach has already answered it correctly to convert the java.util.Date to java.sql.Date and that should solve your issue.

First of all a List cannot have separate dataType for each element in the list as we do have for Tuples . List have only one dataType and if mixed dataTypes are used then the dataType of the list is represented as Any or AnyRef .

I guess you must have created data as below

val list = List(
  List[AnyRef](new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy", Locale.ENGLISH).parse("Sun Jul 31 10:21:53 PDT 2016"), "pm1", 11L: java.lang.Long,"ri1"),
  List[AnyRef](new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy", Locale.ENGLISH).parse("Mon Aug 01 12:57:09 PDT 2016"), "pm3", 5L: java.lang.Long, "ri1"),
  List[AnyRef](new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy", Locale.ENGLISH).parse("Mon Aug 01 01:11:16 PDT 2016"), "pm1", 1L: java.lang.Long, "ri2")
)

val rdd1 = spark.sparkContext.parallelize(list)

which would give

rdd1: org.apache.spark.rdd.RDD[List[AnyRef]]

but in fact its real datatypes are [java.util.Date, String, java.lang.Long, String]

And looking at your other question you must be having problem converting the rdd to dataframe having following schema

val schema =
  StructType(
    StructField("lotStartDate", DateType, false) ::
      StructField("pm", StringType, false) ::
      StructField("wc", LongType, false) ::
      StructField("ri", StringType, false) :: Nil)

What you can do is utilize java.sql.Date api as answered in your other question and then create dataframe as

val rdd1 = sc.parallelize(list).map(lis => Row.fromSeq(new java.sql.Date((lis.head.asInstanceOf[java.util.Date]).getTime)::lis.tail))
val df = sqlContext.createDataFrame(rdd1,schema)

which should give you

+------------+---+---+---+
|lotStartDate|pm |wc |ri |
+------------+---+---+---+
|2016-07-31  |pm1|11 |ri1|
|2016-08-02  |pm3|5  |ri1|
|2016-08-01  |pm1|1  |ri2|
+------------+---+---+---+

I hope the answer is helpful