如何合并这两个数据帧以在 Spark Scala 中生成第三个 dataframe？

Question

由于无法修改 spark scala 中的特定列值，我很难加入这两个 dataframe 视图。 我想我必须以某种方式进行转置/加入，但无法弄清楚。

这是第一个 dataframe：

  var sample_df = Seq(("john","morning","7am"),("john","night","10pm"),("bob","morning","8am"),("bob","night","11pm"),("phil","morning","9am"),("phil","night","10pm")).toDF("person","time_of_day","wake/sleep hour")

这是第二个 dataframe：

  var sample_df2 = Seq(("john","6am","11pm"),("bob","7am","2am"),("phil","8am","1am")).toDF("person","morning_earliest","night_latest")

这是我要生产的结果 dataframe：

  var resulting_df = Seq(("john","morning","7am","6am"),("john","night","10pm","11pm"),("bob","morning","8am","7am"),("bob","night","11pm","2am"),("phil","morning","9am","8am"),("phil","night","10pm","1am")).toDF("person","time_of_day","wake/sleep hour","earliest/latest")

任何帮助将不胜感激！ 谢谢，祝你有美好的一天！

Answer 1

sample_df.createOrReplaceTempView("df1")
sample_df2.createOrReplaceTempView("df2")

spark.sql("""
select person, time_of_day, `wake/sleep hour`, `earliest/latest`
from (
    select person, stack(2, 'morning', morning_earliest, 'night', night_latest) as (time_of_day, `earliest/latest`)
    from df2
) df
join df1
using (time_of_day, person)
""").show()

+------+-----------+---------------+---------------+
|person|time_of_day|wake/sleep hour|earliest/latest|
+------+-----------+---------------+---------------+
|  john|    morning|            7am|            6am|
|  john|      night|           10pm|           11pm|
|   bob|    morning|            8am|            7am|
|   bob|      night|           11pm|            2am|
|  phil|    morning|            9am|            8am|
|  phil|      night|           10pm|            1am|
+------+-----------+---------------+---------------+

Answer 2

val df = sample_df .join(sample_df2,"person") val resulting_df = df.withColumn("earliest/latest", when(col("time_of_day")=== "morning", $"morning_earliest") .otherwise($"night_latest")) .drop($"morning_earliest") .drop($"night_latest") resulting_df.show()