当我使用带有spark / scala的window.partitionBy（）函数时，如何保持partition'number不变？

Question

I have a RDD , the RDD' partition of result changes to 200 when I use window ,can I not change partition when I use window ?

This is my code:

val rdd= sc.parallelize(List(1,3,2,4,5,6,7,8),4)
val result = rdd.toDF("values").withColumn("csum", sum(col("values")).over(Window.partitionBy(col("values")))).rdd
println(result.getNumPartitions + "rdd2")

My input partition is 4, why result partition is 200?

I want my result partition to be also 4.

Is there any cleaner solution?

Answer 1

Note: As mentioned by @eliasah - it's not possible to avoid repartition when using window functions with spark

---

• Why result partition is 200?

Spark doc The default value of spark.sql.shuffle.partitions which Configures the number of partitions to use when shuffling data for joins or aggregations - is 200

• How can I repartition to 4?

You can use:

coalesce(4)

or

repartition(4)

spark doc

coalesce(numPartitions) Decrease the number of partitions in the RDD to numPartitions. Useful for running operations more efficiently after filtering down a large dataset.

repartition(numPartitions) Reshuffle the data in the RDD randomly to create either more or fewer partitions and balance it across them. This always shuffles all data over the network.

Answer 2

(also added this answer to https://stackoverflow.com/a/44384638/3415409 )

I was just reading about controlling the number of partitions when using groupBy aggregation, from https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-performance-tuning-groupBy-aggregation.html , it seems the same trick works with Window, in my code I'm defining a window like

windowSpec = Window \
    .partitionBy('colA', 'colB') \
    .orderBy('timeCol') \
    .rowsBetween(1, 1)

and then doing

next_event = F.lead('timeCol', 1).over(windowSpec)

and creating a dataframe via

df2 = df.withColumn('next_event', next_event)

and indeed, it has 200 partitions. But, if I do

df2 = df.repartition(10, 'colA', 'colB').withColumn('next_event', next_event)

it has 10!