how to add rows to a data frame that are in another data frame by a column in pyspark

Question

I have 2 dfs and i want to get the rows that in the second df to the first one. but I want to add these rows only if the value in the cid column isnt in the first one.

df1
x  y  z  cid
4  8  1  1
7  5  6  2
7  3  5  3

df2
x  y  z  cid
8  4  5  1
1  2  9  2
8  6  4  3
4  5  4  4

result:
x  y  z  cid
4  8  1  1
7  5  6  2
7  3  5  3
4  5  4  4

Answer 1

You can try below code.

from pyspark.sql.functions import *
# Create DataFrame df1
df1 = spark.createDataFrame([(4,8,1,1), (7,5,6,2), (7,3,5,3)], ["x", "y", "z", "cid"])

# Create DataFrame df2
df2 = spark.createDataFrame([(8,4,5,1), (1,2,9,2), (8,6,4,3), (4,5,4,4)], ["x", "y", "z", "cid"])

# Get the values from cid column from df1
col1 = df1.select(collect_set("cid")).collect()[0][0]

# Filter the dataframe df2 where cid values present in df2 but not in df1.
df3 = df2.filter(~df2["cid"].isin(col1))

# Union df1 and df3.
df4 = df1.union(df3)
df4.show()

// Output

+---+---+---+---+
|  x|  y|  z|cid|
+---+---+---+---+
|  4|  8|  1|  1|
|  7|  5|  6|  2|
|  7|  3|  5|  3|
|  4|  5|  4|  4|
+---+---+---+---+

I hope this helps.