Getting error from spark : Number of dynamic partitions created is 1041, which is more than 1000

Question

Hi I am getting below issue in spark.

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Number of dynamic partitions created is 1041, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1041.

I have set below properties in spark but still I am getting this error.

spark.conf.set("hive.exec.dynamic.partition.mode", "nonstrict")
spark.conf.set("hive.exec.dynamic.partition", "true")
spark.conf.set("hive.exec.max.dynamic.partitions","8000")
spark.conf.set("hive.exec.max.dynamic.partitions.pernode","8000")

My Query :

spark.sql("INSERT INTO table_name PARTITION(column_name) SELECT * FROM table_name WHERE year = 2020 and month = 08 and date = 29")

Help me with the solution.

Answer 1

One way to solve this problem would be to check, do you even need these many partitions. Maybe you don't.

This number represents data partitions in data write to database. If it's too high maybe you're setting wrong values for partition key. Try checking that.

One example would be setting now() instead of date+hour or date as value for partition key. Now being a value that changes often, leads to creation of too many partitions.