将 dataframe 拆分为 Scala Spark 中的多个数据帧

Question

I have below JSON file(details) in hadoop. I am able to read this file from hd fs by using SQL Context read json. Then want to split the file into number of files depending on the date and add date to file name (there can be any number of dates in file).

input file Name: details

{"Name": "Pam", "Address": "", "Gender":"F", "Date": "2019-09-27 06:47:57"}
{"Name": "David", "Address": "", "Gender":"M", "Date": "2019-09-27 10:47:56"}
{"Name": "Mike", "Address": "", "Gender":"M", "Date": "2019-09-26 08:48:57"}

expected output files:

file Name 1: details_20190927

{"Name": "Pam", "Address": "", "Gender":"F", "Date": "2019-09-27 06:47:57"}
{"Name": "David", "Address": "", "Gender":"M", "Date": "2019-09-27 10:47:56"}

file Name 2: details_20190926

{"Name": "Mike", "Address": "", "Gender":"M", "Date": "2019-09-26 08:48:57"}

Answer 1

The paths won't be exactly as you have specified them, but you can write the records on different files like this:

import org.apache.spark.sql.functions._;
import spark.implicits._

val parsed = spark.read.json("details.json")
val repartitioned = parsed.repartition(col("Date"))
val withPartitionValue = parsed.withColumn("PartitionValue", date_format(col("Date"),"yyyyMMdd"))
withPartitionValue.write.partitionBy("PartitionValue").json("/my/output/folder")