分区和spark.sql.shuffle.partition中的Spark差异

Question

I am running a spark program with --conf spark.sql.shuffle.partitions=100

Inside application I have the following

Dataset<Row> df_partitioned = df.repartition(df.col("enriched_usr_id"));
df_partitioned = df_partitioned.sortWithinPartitions(df_partitioned.col("transaction_ts"));
df_partitioned.mapPartitions(
    SparkFunctionImpl.mapExecuteUserLogic(), Encoders.bean(Transformed.class));

I have around 5 million users and I want to sort data for every user and execute some logic per user.

My question is does this partition the data into 5 million partitions or 100 partitions and how does the execution work per user.

Answer 1

df.repartition(df.col("enriched_usr_id"))将使用df.repartition(df.col("enriched_usr_id"))将数据划分为100个分区（ spark.sql.shuffle.partitions ），这意味着多个用户将位于同一分区中。