SPARK: dropDuplicates in every partitions only

Question

I want to dropDuplicates in every partitions, not the full DataFrame.

Is that possible with PySpark? Thanks.

Answer 1

import pyspark.sql.functions as f
withNoDuplicates = df.withColumn("partitionID", f.spark_partition_id()).dropDuplicates()

Basically you add a column of the partition id using spark_partition_id and then do the distinct, it will consider different partitions separately