how to replace distinct() with reducebykey

Question

I have a scenario where the below code overall take more than 10 hours for >2 Billion records. even i tried with 35 instance of the i3 cluster but still the performance was bad. I am looking for an option to replace distinct() with reduceByKey() and to get suggestion to improve the performance...

    val df = spark.read.parquet(out)
      
     val df1 = df.
    select($"ID", $"col2", $"suffix",
   $"date", $"year", $"codes")

   val df2 = df1.
    repartition(
      List(col("ID"), col("col2"), col("suffix"), col("date"),
        col("year"), col("codes")): _*
    ).distinct()
      
       val df3 = df2.withColumn("codes", expr("transform(codes, (c,s) -> (d,s) )"))
    
       df3.createOrReplaceTempView("df3")
    
       val df4 = spark.sql(
         """SELECT
               ID, col2, suffix
               d.s as seq,
               d.c as code,
               year,date
               FROM
                df3
                 LATERAL VIEW explode(codes) exploded_table as d
                 """)
    
       df4.
         repartition(
           600,
           List(col("year"), col("date")): _*).
         write.
         mode("overwrite").
         partitionBy("year", "date").
         save(OutDir)

Answer 1

I think distinct() is implemented with reduceByKey (reduce), but if you want to implement it by yourself, you could do something

val array=List((1,2),(1,3),(1,5),(1,2),(2,2),(2,2),(3,2),(3,2),(4,1),(1,3))
val pairRDD=session.sparkContext.parallelize(array)
val distinctResult=pairRDD.map(x => (x, null)).reduceByKey((x, _) => x)