如何在Pyspark中比较两个数据帧

Question

c = df[df['CUSTOMER_EMAIL_ID'].isin(d.CUSTOMER_EMAIL_ID)]

如何在PySpark中编写相同的表达式？

Answer 1

If you're asking "give me all the rows from df where the CUSTOMER_EMAIL_ID field has a matching value from the CUSTOMER_EMAIL_ID field in d ", then I think your question can be answered using a semi join , specifically:

c = df.join(b, 'CUSTOMER_EMAIL_ID', 'leftsemi')

A left (right) semi join can be thought of conceptually as a inner join followed by dropping the right (left) columns.