Spark DataFrame zipWithIndex

Question

I am using a DataFrame to read in a .parquet files but than turning them into an rdd to do my normal processing I wanted to do on them.

So I have my file:

val dataSplit = sqlContext.parquetFile("input.parquet")
val convRDD = dataSplit.rdd 
val columnIndex = convRDD.flatMap(r => r.zipWithIndex)

I get the following error even when I convert from a dataframe to RDD:

:26: error: value zipWithIndex is not a member of org.apache.spark.sql.Row

Anyone know how to do what I am trying to do, essentially trying to get the value and the column index.

I was thinking something like:

val dataSplit = sqlContext.parquetFile(inputVal.toString)
val schema = dataSplit.schema
val columnIndex = dataSplit.flatMap(r => 0 until schema.length

but getting stuck on the last part as not sure how to do the same of zipWithIndex.

Answer 1

You can simply convert Row to Seq :

convRDD.flatMap(r => r.toSeq.zipWithIndex)

Important thing to note here is that extracting type information becomes tricky. Row.toSeq returns Seq[Any] and resulting RDD is RDD[(Any, Int)] .