如何在Spark中使用Dataframe将数据加载到Product case类中

Question

I have a text file and has data like below:

productId|price|saleEvent|rivalName|fetchTS 
123|78.73|Special|VistaCart.com|2017-05-11 15:39:30 
123|45.52|Regular|ShopYourWay.com|2017-05-11 16:09:43 
123|89.52|Sale|MarketPlace.com|2017-05-11 16:07:29 
678|1348.73|Regular|VistaCart.com|2017-05-11 15:58:06 
678|1348.73|Special|ShopYourWay.com|2017-05-11 15:44:22 
678|1232.29|Daily|MarketPlace.com|2017-05-11 15:53:03 
777|908.57|Daily|VistaCart.com|2017-05-11 15:39:01 

I have to find minimum price of a product across websites, eg my output should be like this:

productId|price|saleEvent|rivalName|fetchTS 
123|45.52|Regular|ShopYourWay.com|2017-05-11 16:09:43 
678|1232.29|Daily|MarketPlace.com|2017-05-11 15:53:03 
777|908.57|Daily|VistaCart.com|2017-05-11 15:39:01 

I am trying like this:

case class Product(productId:String, price:Double, saleEvent:String, rivalName:String, fetchTS:String)

val cDF = spark.read.text("/home/prabhat/Documents/Spark/sampledata/competitor_data.txt")
val (header,values) = cDF.collect.splitAt(1)
values.foreach(x => Product(x(0).toString, x(1).toString.toDouble, 
x(2).toString, x(3).toString, x(4).toString))

Getting exception while running last line:

 java.lang.ArrayIndexOutOfBoundsException: 1
 at org.apache.spark.sql.catalyst.expressions.GenericRow
 .get(rows.scala:174)
 at org.apache.spark.sql.Row$class.apply(Row.scala:163)
 at 
 org.apache.spark.sql.catalyst.expressions.GenericRow
 .apply(rows.scala:166
 )
 at $anonfun$1.apply(<console>:28)
 at $anonfun$1.apply(<console>:28)
 at scala.collection.IndexedSeqOptimized$class.foreach
 (IndexedSeqOptimized.scala:33)
 at 
 scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
 ... 49 elided

Priting value in values :

scala> values
res2: **Array[org.apache.spark.sql.Row]** = ` 
Array([123|78.73|Special|VistaCart.com|2017-05-11 15:39:30 ], 
[123|45.52|Regular|ShopYourWay.com|2017-05-11 16:09:43 ], 
[123|89.52|Sale|MarketPlace.com|2017-05-11 16:07:29 ], 
[678|1348.73|Regular|VistaCart.com|2017-05-11 15:58:06 ], 
[678|1348.73|Special|ShopYourWay.com|2017-05-11 15:44:22 ], 
[678|1232.29|Daily|MarketPlace.com|2017-05-11 15:53:03 ], 
[777|908.57|Daily|VistaCart.com|2017-05-11 15:39:01 ]`
scala> 

I am able to understand that I need to split("|") .

scala> val xy = values.foreach(x => x.toString.split("|").toSeq)
xy: Unit = ()

So after splitting its giving me Unit class, ie void, so unable to load values into the Product case class. How can I load this Dataframe to Product case class? I dont want to use Dataset for now, although Dataset is type safe.

I'm using Spark 2.3 and Scala 2.11.

Answer 1

The issue is due to split taking a regex, which means you need to use "\\\\|" instead of a single "|" . Also, the foreach need to be changed to map to actually give a return value, ie:

val xy = values.map(x => x.toString.split("\\|"))

---

However, a better approach would be to read the data as a csv file with | separators. In this way you do not need to treat the header in a special way and by inferring the column types there is no need to make any convertions (here I changed fetchTS to a timestamp):

case class Product(productId: String, price: Double, saleEvent: String, rivalName: String, fetchTS: Timestamp)

val df = spark.read
  .option("header", "true")
  .option("inferSchema", "true")
  .option("sep", "|")
  .csv("/home/prabhat/Documents/Spark/sampledata/competitor_data.txt")
  .as[Product]

The final line will convert the dataframe to use the Product case class. If you want to use it as an RDD instead, simply add .rdd in the end.

After this is done, use groupBy and agg to get the final results.