如何在 Spark Scala 中将 RDD 转换为 DF？

Question

I am new in spark. And I am trying to convert below RDD to dataframe but not succeed

val customerRDD = sc.textFile("file:///home/hduser/data//customer.txt") //custId,CustName,CustEmail,CustPhone //1,ABC,abc@gmail.com,+199240242234

Here I am trying to use customerRDD.toDF() method but not working

Also I have tried with createDataFrame() method but not able to get the idea

Does anyone can help How can I convert RDD to DF here?

Thanks

Answer 1

An odd way of doing things these days, but if you must use an RDD to read a file with a header, then consult this https://sparkbyexamples.com/apache-spark-rdd/spark-load-csv-file-into-rdd/ and note specifically:

• Skip the header of each file (can be seen)
• Extract the columns yourself via map (can be seen)

Look at this for creating DF from RDD with schema using Structs, see https://sparkbyexamples.com/apache-spark-rdd/convert-spark-rdd-to-dataframe-dataset . You can

• create a schema programmatically for a DF from RDD via createDataFrame()
• or use default schema with implicits