如何在Hive上对未分区的表进行分区？

Question

Given a table with 360 days of data, we want to partition it by date to improve performance. Do we need to use following SELECT command for each date? Any more efficient way to do this?

INSERT INTO TABLE <new_table> Partition (dt='2015-07-01')
SELECT * from <table> WHERE dt='2015-07-01'

Answer 1

If your new table is partitioned by dt (date), you should use Dynamic Partition . You dont need to specify the specific partition (in this case date). In this way Hive realize all different dates and it makes the partitions automatically.

Remember set these flags:

set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;

Answer 2

First make your table:

create  db.my_table(column1 int, column2 string,
                     -- ...
)
comment 'I like paritioned tables'
partitioned by(dt string)
location '/path/to/file';

Now you can load the data into dt partitions:

insert overwrite into table db.my_table partition (dt) select * from other_table;