使用 Scala Spark 将文本文件转换为数据框

Question

I have a text file from S3 (actually multiple .gz files) and I wrote the code below

val text = sc.textFile(path)
val df_text = text.map(row => row.split(",")).toDF()

But the result is like that

+--------------------+
|               value|
+--------------------+
|[id, member_id, l...|
|[1077501, 1296599...|
|[1077430, 1314167...|
|[1077175, 1313524...|
|[1076863, 1277178...|
|[1075358, 1311748...|
|[1075269, 1311441...|
+--------------------+

I can't read it like "val df = spark.read.format("csv").option("header", "true").load(path)" because when I read it like that it can't find header:

+-----------+-----------+-----------+
|1077430    |1356730    |4525526    |...
+-----------+-----------+-----------+
|   41173430|    1356730|    1456430|...
|   10237430|    1356660|    1463750|...
+-----------+-----------+-----------+

How can I make it a proper DataFrame?

Answer 1

In spark 2.4.0 with scala 2.12.8.

It's very easy:

val spark: SparkSession = SparkSession
      .builder
      .master("local[*]")
      .getOrCreate
val sc = spark.sparkContext

val myGZs= sc
      .textFile("s3://route//*.gz")
      .map(parseToObject)
      .filter(obj => obj != null)

val myGZsDF = spark.createDataFrame(myGZs)
myGZsDF.printSchema()

where parseToObject is a function like:

val parseToObject = (row: String) => {
   if (row is header) { //program this
      null
   }
   val split_row = row.split(",")
   Subscription(split_row[0].toLong, split_row[1].toLong ...)
}

An example of Subscription case class:

case class Subscription(id: Long, memberId: Long ...)

Both map and filter are narrow transformations!

EDIT:

Have also seen this link from @kev on how to read multiple GZ files and convert it to a DF. Beware of the extension, it MUST be .gz .

Hope this helps. Let me know if you have any problem. Tomás.