在 Pyspark 数据帧上左连接和应用案例逻辑

Question

I am learning to code PySpark. I am able join two dataframes by building SQL like views on top them using .createOrReplaceTempView() and get the output I want. However I want to learn how to do the same by operating directly on the dataframe instead of creating views .

This is my code

df1.createOrReplaceTempView('left_table')
df2.createOrReplaceTempView('right_table')

    spark.sql('''
    select
    l.*,
    CASE WHEN r.id IS NULL THEN current_timestamp() ELSE r.timestamp END ts,
    from
    left_table l 
    left join 
    right_table r
    on l.id = r.id 
    ''').show()

For matching id I want the timestamp column to be taken from the right table. For id that is available only in left table I want to use system timestamp using current_timestamp() for the final column value.

How do i achieve this by operating directly on dataframes df1 and df2 instead of building views?

Answer 1

You can do a left join and then coalesce the NULL timestamps with the current timestamp:

import pyspark.sql.functions as F

df1.join(df2, 'id', 'left') \
   .drop(*[col for col in df2.columns if col != 'timestamp']) \
   .withColumn('timestamp', F.coalesce(F.col('timestamp'), F.current_timestamp()))