想根据Pyspark中的两个列加入dataframe

Question

I am using pyspark in databricks. I have two dataframes, df1 and df2. I want to left join df1 with df2. However based on condition on df2's columns.

From df1, it checks if df1.ID is present in df2.A, if it is present, it takes the row value, else if it is Null, then it checks df2.B, if it is same as df.ID, it keeps it.

df1

| ID       | 
| -------- | 
| aaa      | 
| bbb      | 
| ccc      | 
| ddd      | 
| eee      | 

df2

|     A    |     B    |     C    |
| -------- | -------- | -------- |
| aaa      |    aaa   | 23       |
| eee      |    bbb   | 32       |
| Null     |    ccc   | 45       |
| Null     |    ddd   | 76       |

Output

| ID       |      A    |     B    |     C    |
| -------- | --------  | -------- | -------- | 
| aaa      | aaa       |    aaa   | 23       |
| bbb      |           |          |          |
| ccc      | Null      |    ccc   | 45       | 
| ddd      | Null      |    ddd   | 76       | 
| eee      |  eee      |    bbb   | 32       |

I tried following but it is not giving me correct results:

join_conditions = [
    df1.ID == df2.A,
    (df1.ID  == df2.B) | (df2.A.isNull())
]

df3 = df1.join(df2, join_conditions,"left")

Answer 1

Use conditions df2.A.isNotNull() & (df1.ID == df2.A) and df2.A.isNull() & (df1.ID == df2.B) and separate them with |:

import pyspark.sql.functions as F
df1 = spark.createDataFrame(data=[["aaa"],["bbb"],["ccc"],["ddd"],["eee"]], schema=["ID"])
df2 = spark.createDataFrame(data=[["aaa","aaa","23"],["eee","bbb","32"],[None,"ccc","45"],[None,"ddd","76"]], schema=["A","B","C"])

result_df = df1.join(df2,
                     (
                         (df2.A.isNotNull() & (df1.ID == df2.A))
                         |
                         (df2.A.isNull() & (df1.ID == df2.B))
                     ),
                     how="left"
                     )

result_df.show()
+---+----+----+----+
| ID|   A|   B|   C|
+---+----+----+----+
|aaa| aaa| aaa|  23|
|bbb|null|null|null|
|ccc|null| ccc|  45|
|ddd|null| ddd|  76|
|eee| eee| bbb|  32|
+---+----+----+----+