Reading Partitioned HIVE table in SPARK SQL

Question

I am trying to read the HIVE table has 2 years of data and it is partitioned on day, Reading it through SPARK-SQL takes more than 10 mins even after filter applied for 6 months data.

Any idea how to improve, i have tried with DISTRIBUTE BY clause in Hive QL

Answer 1

You can enable partitioning pruning by setting property:

spark.sql.hive.metastorePartitionPruning=true

This will filter the partitions