从每个分组的 Spark 数据框中选择特定行

Question

我有一个这样的数据框：

+-----------------+------------------+-----------+--------+---+
| conversation_id |   message_body   | timestamp | sender |   |
+-----------------+------------------+-----------+--------+---+
| A               | hi               | 9:00      | John   |   |
| A               | how are you?     | 10:00     | John   |   |
| A               | can we meet?     | 10:05     | John   | * |
| A               | not bad          | 10:30     | Steven | * |
| A               | great            | 10:40     | John   |   |
| A               | yeah, let's meet | 10:35     | Steven |   |
| B               | Hi               | 12:00     | Anna   | * |
| B               | Hello            | 12:05     | Ken    | * |
+-----------------+------------------+-----------+--------+---+

对于每个对话，我希望第一个发件人的第一个块中的最后一条消息和第二个发件人的第一条消息。 我用星号标记了它们。

我的一个想法是为第一个用户分配 0，为第二个用户分配 1。

理想情况下，我想要这样的东西：

+-----------------+---------+------------+--------------+---------+------------+----------+
| conversation_id | sender1 | timestamp1 |   message1   | sender2 | timestamp2 | message2 |
+-----------------+---------+------------+--------------+---------+------------+----------+
| A               | John    | 10:05      | can we meet? | Steven  | 10:30      | not bad  |
| B               | Anna    | 12:00      | Hi           | Ken     | 12:05      | Hello    |
+-----------------+---------+------------+--------------+---------+------------+----------+

我怎么能在 Spark 中做到这一点？

Answer 1

有趣的问题出现了。

• 10:35 至 10:45 修改
• 使用前导 0 格式，例如 09:00 而不是 9:00
• 您将需要相应地使用自己的数据类型，这只是演示了所需的方法

• 在 DataBricks Notebook 中完成

  import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.functions._ import spark.implicits._ val df = Seq(("A", "hi", "09:00", "John"), ("A", "how are you?", "10:00", "John"), ("A", "can we meet?", "10:05", "John"), ("A", "not bad", "10:30", "Steven"), ("A", "great", "10:40", "John"), ("A", "yeah, let's meet", "10:45", "Steven"), ("B", "Hi", "12:00", "Anna"), ("B", "Hello", "12:05", "Ken") ).toDF("conversation_id", "message_body", "timestampX", "sender") // Get rank, 1 is who were initiates conversation, the other values can be used to infer relationships // Note no @Transient required here with Window val df2 = df.withColumn("rankC", row_number().over(Window.partitionBy($"conversation_id").orderBy($"timestampX".asc))) // A value <> 1 is the first message of second Sender. // The 1 value of this is the last message of first "block" val df3 = df2.select('conversation_id as "df3_conversation_id", 'sender as "df3_sender", 'rankC as "df3_rank") // To avoid pipelined renaming issues that occur val df3a = df3.groupBy("df3_conversation_id", "df3_sender").agg(min("df3_rank") as "rankC2").filter("rankC2 != 1") //JOIN the values with some smarts. Some odd errors in Spark thru pipe-lining gotten. Need to drop pipelined row(), ranking etc. val df4 = df3a.join(df2, (df3a("df3_conversation_id") === df2("conversation_id")) && (df3a("rankC2") === df2("rankC") + 1)).drop("rankC").drop("rankC2") val df4a = df3a.join(df2, (df3a("df3_conversation_id") === df2("conversation_id")) && (df3a("rankC2") === df2("rankC"))).drop("rankC").drop("rankC2") // The get other missing data, could have all been combined but done in steps for simplicity. Just a simple final JOIN and you ahve the answer. val df5 = df4.join(df4a, (df4("df3_conversation_id") === df4a("df3_conversation_id"))) df5.show(false)

返回：

输出不会在这里完全格式化，在 REPL 中运行它以查看标题。

 |B                  |Ken       |B              |Hi          |12:00     |Anna  |B                  |Ken       |B              |Hello       |12:05     |Ken   |
 |A                  |Steven    |A              |can we meet?|10:05     |John  |A                  |Steven    |A              |not bad     |10:30     |Steven|

您可以进一步操作数据，繁重的工作现已完成。 Catalyst Optimizer 在编译等方面存在一些问题，所以这就是我以这种方式工作的原因。