Pyspark，如何 append dataframe 但從特定的一個中刪除重復項

Question

我每天從源中檢索一次數據，但由於某些延遲，我需要檢索比上一次檢索的最新數據更早的數據。 這會導致一些重疊，我想要實現的是刪除舊 dataframe 中具有新 dataframe 中的時間戳的行，以便我只保留最近檢索到的信息。

數據示例：

df_old.show()

+-------------------+------------------+------------------+
|              index|                 A|                 B|
+-------------------+------------------+------------------+
|2013-01-01 00:00:00| 6.251379599223777| 10.23320055553287|
|2013-01-01 00:10:00| 6.245690342672945| 10.22296550603164|
|2013-01-01 00:20:00|6.2534029157968956|10.221452136599193|
|2013-01-01 00:30:00| 6.247532408988978|10.212423634472028|
|2013-01-01 00:40:00| 6.253508510989639|10.194494950388954|
|2013-01-01 00:50:00| 6.247517363773414|10.200814690766375|
|2013-01-01 01:00:00|  6.25381864046542|10.192425005184585|
|2013-01-01 01:10:00| 6.250060498528904|10.181246688945123|
|2013-01-01 01:20:00| 6.254461614739839| 10.18021442155982|
|2013-01-01 01:30:00| 6.233226501275796|10.180681886095698|
|2013-01-01 01:40:00| 6.252799353320566|10.169008765187861|
|2013-01-01 01:50:00| 6.248423707837854| 10.16567354928804|
|2013-01-01 02:00:00| 6.253744374163072|10.161773904107136|
|2013-01-01 02:10:00| 6.238242597088755|10.151641862402213|
+-------------------+------------------+------------------+


df_new.show()

+-------------------+------------------+------------------+
|              index|                 A|                 B|
+-------------------+------------------+------------------+
|2013-01-01 01:30:00| 7                | 20               |
|2013-01-01 01:40:00| 7                | 20               |
|2013-01-01 01:50:00| 7                | 20               |
|2013-01-01 02:00:00| 7                | 20               |
|2013-01-01 02:10:00| 7                | 20               |
|2013-01-01 02:20:00|  6.24546611958182| 10.14886792741417|
|2013-01-01 02:30:00| 6.240802043802097| 10.15267232231782|
|2013-01-01 02:40:00| 6.249921473522189|10.139161473568803|
|2013-01-01 02:50:00|6.2219054718011515| 10.11521891469772|
|2013-01-01 03:00:00| 6.247084671443932|10.088592826542145|
|2013-01-01 03:10:00|  6.24950717588649|10.065343892142995|
+-------------------+------------------+------------------+

我想要實現的是這個結果，其中只保留了新 df 的重疊結果：

df_combined.show()

+-------------------+------------------+------------------+
|              index|                 A|                 B|
+-------------------+------------------+------------------+
|2013-01-01 00:00:00| 6.251379599223777| 10.23320055553287|
|2013-01-01 00:10:00| 6.245690342672945| 10.22296550603164|
|2013-01-01 00:20:00|6.2534029157968956|10.221452136599193|
|2013-01-01 00:30:00| 6.247532408988978|10.212423634472028|
|2013-01-01 00:40:00| 6.253508510989639|10.194494950388954|
|2013-01-01 00:50:00| 6.247517363773414|10.200814690766375|
|2013-01-01 01:00:00|  6.25381864046542|10.192425005184585|
|2013-01-01 01:10:00| 6.250060498528904|10.181246688945123|
|2013-01-01 01:20:00| 6.254461614739839| 10.18021442155982|
|2013-01-01 01:30:00| 7                | 20               |
|2013-01-01 01:40:00| 7                | 20               |
|2013-01-01 01:50:00| 7                | 20               |
|2013-01-01 02:00:00| 7                | 20               |
|2013-01-01 02:10:00| 7                | 20               |
|2013-01-01 02:20:00|  6.24546611958182| 10.14886792741417|
|2013-01-01 02:30:00| 6.240802043802097| 10.15267232231782|
|2013-01-01 02:40:00| 6.249921473522189|10.139161473568803|
|2013-01-01 02:50:00|6.2219054718011515| 10.11521891469772|
|2013-01-01 03:00:00| 6.247084671443932|10.088592826542145|
|2013-01-01 03:10:00|  6.24950717588649|10.065343892142995|
+-------------------+------------------+------------------+

是否有任何簡單的內置函數可以實現此結果？

Answer 1

使用outer連接。

df1.join(df2, ['index'], 'outer') \
  .select('index', coalesce(df2.A, df1.A), coalesce(df2.B, df1.B)).toDF('index', 'A', 'B') \
  .orderBy('index').show(20, False)

+-------------------+------------------+------------------+
|index              |A                 |B                 |
+-------------------+------------------+------------------+
|2013-01-01 00:00:00|6.251379599223777 |10.23320055553287 |
|2013-01-01 00:10:00|6.245690342672945 |10.22296550603164 |
|2013-01-01 00:20:00|6.2534029157968956|10.221452136599193|
|2013-01-01 00:30:00|6.247532408988978 |10.212423634472028|
|2013-01-01 00:40:00|6.253508510989639 |10.194494950388954|
|2013-01-01 00:50:00|6.247517363773414 |10.200814690766375|
|2013-01-01 01:00:00|6.25381864046542  |10.192425005184585|
|2013-01-01 01:10:00|6.250060498528904 |10.181246688945123|
|2013-01-01 01:20:00|6.254461614739839 |10.18021442155982 |
|2013-01-01 01:30:00|7.0               |20.0              |
|2013-01-01 01:40:00|7.0               |20.0              |
|2013-01-01 01:50:00|7.0               |20.0              |
|2013-01-01 02:00:00|7.0               |20.0              |
|2013-01-01 02:10:00|7.0               |20.0              |
|2013-01-01 02:20:00|6.24546611958182  |10.14886792741417 |
|2013-01-01 02:30:00|6.240802043802097 |10.15267232231782 |
|2013-01-01 02:40:00|6.249921473522189 |10.139161473568803|
|2013-01-01 02:50:00|6.2219054718011515|10.11521891469772 |
|2013-01-01 03:00:00|6.247084671443932 |10.088592826542145|
|2013-01-01 03:10:00|6.24950717588649  |10.065343892142995|
+-------------------+------------------+------------------+