如何使用Pig中的HCatlog对Hive Metastore使用压缩技术？

Question

I have some pig script which takes input from normal text files using PigStorage(). I want to load and store data from hive metastore, for that I have used HcatLoader() and HcatStorage() from Hcatalog. Can someone tell me how can i store and load the compress hive data in pig.

Answer 1

Pig generally knows how to load compressed data automatically, if it was compressed using gzip or bzip2. For LZO, you will have to have that enabled on your cluster.

To store data in compressed form, you can put this in your script:

SET mapred.output.compress true;
SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;

This will cause your output to be compressed using gzip.

Answer 2

Part of the charter of HCatalog is for consumer to be completely unaware of storage concerns (like compression or formats). If the underlying storage is uncompressed at first then compressed later, you wouldn't have to rewrite your scripts to make sure that you're reading compressed data.

Having said that... I don't think compression support is implemented in HCatalog yet. HCatalog Roadmap - Written a long time ago... but has "compression" in envisioned future features.

My guess is that you'll have to resort to using the HiveStorage class instead of HCatalog.

Disclaimer: I could also be completely mistaken about this, but all evidence I've been able to find seems to suggest that compression is not implemented in HCatalog.