Pyspark - 从两个不同的数据帧中减去列

Question

I have two dataframes below:

df1:                                             df2:
  +------------+------------+-----------+        +-----------+-------------+-----------+    
  |  date      |Advertiser  |Impressions|        |   date    |Advertiser   |Impressions|
  +------------+------------+-----------+        +-----------+-------------+-----------+
  |2020-01-08  |b           |50035      |        | 2020-01-07|b            |10000      |
  |2020-01-08  |c           |70000      |        | 2020-01-07|c            |25260      |
  +------------+------------+-----------+        +-----------+-------------+-----------+ 
  

I would like to do df1(Impressions) - df2(Impressions), and save it to a new dataframe df3.

  +------------+------------+----------------+               
  |  date      |Advertiser  |diff Impressions|       
  +------------+------------+----------------+        
  |2020-01-08  |b           |40035           |        
  |2020-01-08  |c           |44740           |          
  +------------+------------+----------------+

Answer 1

You can join the two dataframes using the Advertiser column and make appropriate selections:

df3 = df1.join(df2, 'Advertiser').select(
    df1.date, 
    'Advertiser', 
    (df1.Impressions - df2.Impressions).alias('diff Impressions')
)

df3.show()
+----------+----------+----------------+
|      date|Advertiser|diff Impressions|
+----------+----------+----------------+
|2020-01-08|         b|           40035|
|2020-01-08|         c|           44740|
+----------+----------+----------------+