To store data in a hcatalog table, the table must be empty. What about incremetal loading?

Question

as far as I'm concerned there is no support for incremental loading in Hadoop. I'm playing a bit with pig and hive. For testing purposes I just read existing values from table in hcatalog, union it with new data, and than create new table in hcatalog, insert data. Than it needs to maintained with hive to delete old table, rename new one etc...

Other solution is to add new data to a partition in existing table.

Generaly - can you suggest BEST solution for incremental loading?

Regards

Pawel

Answer 1

Adding partition data to Hive/HCatalog table is BEST solution for incremental updates. Further you can have static and dynamic partitions.