蜂巢：从现有分区表创建新表

Question

I'm using Amazon's Elastic MapReduce and I have a hive table created based on a series of log files stored in Amazon S3 and split in folders by day like so:

• data/day=2011-09-01/log_file.tsv
• data/day=2011-09-02/log_file.tsv

I am currently trying to create an additional table which filters out some unwanted activity in these log files but I can't figure out how to do this and keep getting errors such as:

FAILED: Error in semantic analysis: need to specify partition columns because the destination table is partitioned.

If my initial table create statement looks something like this:

CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
... fields ...
)
PARTITIONED BY ( DAY STRING )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 's3://bucketname/data/';

That initial table works fine and I've been able to query it with no problems.

How then should I create a new table that shares the structure of the previous one but simply filters out data? This doesn't seem to work.

CREATE EXTERNAL TABLE IF NOT EXISTS table2 LIKE table1;

FROM table1
INSERT OVERWRITE TABLE table2
SELECT * WHERE
col1 = '%somecriteria%' AND
more criteria...
;

As I've stated above, this returns:

FAILED: Error in semantic analysis: need to specify partition columns because the destination table is partitioned.

Thanks!

Answer 1

This always works for me:

CREATE EXTERNAL TABLE IF NOT EXISTS table2 LIKE table1;
INSERT OVERWRITE TABLE table2 PARTITION (day) SELECT col1, col2, ..., day FROM table1;
ALTER TABLE table2 RECOVER PARTITIONS;

Notice that I've added 'day' as a column in the SELECT statement. Also notice that there is an ALTER TABLE line which is necessary for Hive to become aware of the partitions that were newly created in table2.

Answer 2

I have never used the like option.. so thanks for showing me that. Will that actually create all of the partitions that the first table has as well? If not, that could be the issue. You could try using dynamic partitions :

create external table if not exists table2 like table1;
insert overwrite table table2 partition(part) select col1, col2 from table1;

Might not be the best solution, as I think you have to specify your columns in the select clause (as well as the partition column in the partition clause ).

And, you must turn on dynamic partitioning.

I hope this helps.