当从HDFS手动删除分区数据时，如何在Hive中更新分区元数据

Question

What is the way to automatically update the metadata of Hive partitioned tables?

If new partition data's were added to HDFS (without alter table add partition command execution) . then we can sync up the metadata by executing the command 'msck repair'.

What to be done if a lot of partitioned data were deleted from HDFS (without the execution of alter table drop partition commad execution).

What is the way to syncup the Hive metatdata?

Answer 1

EDIT : Starting with Hive 3.0.0 MSCK can now discover new partitions or remove missing partitions (or both) using the following syntax :

MSCK [REPAIR] TABLE table_name [ADD/DROP/SYNC PARTITIONS]

This was implemented in HIVE-17824

---

As correctly stated by HakkiBuyukcengiz , MSCK REPAIR doesn't remove partitions if the corresponding folder on HDFS was manually deleted, it only adds partitions if new folders are created .

Extract from offical documentation :

In other words, it will add any partitions that exist on HDFS but not in metastore to the metastore.

This is what I usually do in the presence of external tables if multiple partitions folders are manually deleted on HDFS and I want to quickly refresh the partitions :

• Drop the table ( DROP TABLE table_name ) (dropping an external table does not delete the underlying partition files)
• Recreate the table ( CREATE EXTERNAL TABLE table_name ... )
• Repair it ( MSCK REPAIR TABLE table_name )

Depending on the number of partitions this can take a long time. The other solution is to use ALTER TABLE DROP PARTITION (...) for each deleted partition folder but this can be tedious if multiple partitions were deleted.

Answer 2

尝试使用

MSCK REPAIR TABLE <tablename>;

Answer 3

Ensure the table is set to external, drop all partitions then run the table repair:

alter table mytable_name set TBLPROPERTIES('EXTERNAL'='TRUE')
alter table mytable_name drop if exists partition (`mypart_name` <> 'null');
msck repair table mytable_name;

If msck repair throws an error, then run hive from the terminal as:
hive --hiveconf hive.msck.path.validation=ignore
or set hive.msck.path.validation=ignore;