如果行与另一个 DataFrame 中的 ID 匹配并且时间戳低于其他帧时间戳，如何过滤 Scala Spark DataFrame

Question

I want to filter out entries in a DataFrame of message events based on when they were edited. I have a DataFrame that has the message events, and another DataFrame that represents when/if they were edited. The deletion should delete rows in the message table if they have a matching index in the edited table AND if the timestamp in the message table is below the corresponding edit event.

The Edited DataFrame is:

+----------+-------------------+
| timestamp|index              |
+----------+-------------------+
|1556247980|                 78|
|1558144430|                 87|
|1549964820|                 99|
+----------+-------------------+

The Message DataFrame is:

+-------------------+--------------------+------------------+--------------------+
|index              |  commonResponseText|publishedTimestamp|  commonResponseText|
+-------------------+--------------------+------------------+--------------------+
|                 78|Voluptatem enim a...|        1556247974|Voluptatem enim a...|
|                 87|Ut enim enim sunt...|        1558144420|Ut enim enim sunt...|
|                 99|Et est perferendi...|        1549964815|Et est perferendi...|
|                 78|Voluptatem porro ...|        1556248000|Voluptatem porro ...|
|                 87|Atque quod est au...|        1549965000|Atque quod est au...|
+-------------------+--------------------+------------------+--------------------+

I want the result to be:

+-------------------+--------------------+------------------+--------------------+
|commonResponseIndex|  index             |publishedTimestamp|  commonResponseText|
+-------------------+--------------------+------------------+--------------------+
|                 78|Voluptatem porro ...|        1556248000|Voluptatem porro ...|
|                 87|Atque quod est au...|        1549965000|Atque quod est au...|
+-------------------+--------------------+------------------+--------------------+

Thanks for the help!

Answer 1

You can aggregate your message table, join it with edited table and filter

import pyspark.sql.functions as F
# Test dataframe
tst=sqlContext.createDataFrame([('A',2),('B',2),('A',2),('A',3),('B',4),('A',2),('B',2),('c',9)],schema=("id","count"))
tst1 = sqlContext.createDataFrame([('A',4),('B',1)],schema=("id","val"))
# Aggregate and join
tst_g=tst.groupby('id').agg(F.max('count').alias('count'))
tst_j= tst_g.join(tst1,tst_g.id==tst1.id,'left')
# Filter result
tst_f = tst_j.where((F.col('count')>=F.col('val'))|(F.col('val').isNull()))

The result is:

tst_j.show()

+---+-----+----+----+
| id|count|  id| val|
+---+-----+----+----+
|  c|    9|null|null|
|  B|    4|   B|   1|
|  A|    3|   A|   4|
+---+-----+----+----+
 tst_f.show()
+---+-----+----+----+
| id|count|  id| val|
+---+-----+----+----+
|  c|    9|null|null|
|  B|    4|   B|   1|
+---+-----+----+----+

Finally, you can drop the irrelevant columns.

If you need the full data, then you can join the update table with message table and do the same. If the update table is small, then consider a broadcast join for performance reason.

# Approach to join with full table
# Test dataframe
tst=sqlContext.createDataFrame([('A',2),('B',2),('A',2),('A',3),('B',4),('A',2),('B',2),('c',9)],schema=("id","count"))
tst1 = sqlContext.createDataFrame([('A',4),('B',1)],schema=("id","val"))
#%%
# join with the full table
tst_j= tst.join(tst1,tst.id==tst1.id,'left')
# Filter result
tst_f = tst_j.where((F.col('count')>=F.col('val'))|(F.col('val').isNull()))

Hint: if you dont want two id columns in your result you can change the join syntax as tst.join(tst1,on="id",how='left')

Answer 2

Here's what I ended up doing:

val editedDF = Seq(("A",3),("B",3)).toDF("id","timestamp")
val messageDF = Seq(("A",2),("B",2),("A",2),("A",3),("B",4),("A",2),("B",2),("c",9)).toDF("id","timestamp")

Finally I used this join:

    // Filter out the edited meesages.
    val editedFilteredDF  = messageDF.join(editedDF,
(editedDF("id") === messageDF("id")) && (editedDF("timestamp") > messageDF("timestamp")),
joinType="left_anti")

The result:

 editedFilteredDF.show()
+---+---------+
| id|timestamp|
+---+---------+
|  A|        3|
|  B|        4|
|  c|        9|
+---+---------+