Spark 在 S3 中创建额外的分区列

Question

I am writing a dataframe to s3 as shown below. Target location: s3://test/folder

   val targetDf = spark.read.schema(schema).parquet(targetLocation)
    val df1=spark.sql("select * from sourceDf")
    val df2=spark.sql(select * from targetDf)
/*    
for loop over a date range to dedup and write the data to s3
union dfs and run a dedup logic, have omitted dedup code and for loop
*/
    val df3=spark.sql("select * from df1 union all select * from df2")
    df3.write.partitionBy(data_id, schedule_dt).parquet("targetLocation")

Spark is creating extra partition column on write like shown below:

Exception in thread "main" java.lang.AssertionError: assertion failed: Conflicting partition column names detected:

Partition column name list #0: data_id, schedule_dt
Partition column name list #1: data_id, schedule_dt, schedule_dt

EMR optimizer class is enabled while writing, I am using spark 2.4.3 Please let me know what could be causing this error.

Thanks Abhineet

Answer 1

You should give 1 extra column apart from partitioned columns. Can you please try

val df3=df1.union(df2)

instead of

val df3=spark.sql("select data_id,schedule_dt from df1 union all select data_id,schedule_dt from df2")