为什么我的avro输出文件在我的养猪工作中是如此之小而如此之多？

Question

I'm running a pig script that does a series of joins and write using AvroStorage()

All is running well, and I am getting the data that I want... but it is being written to 845 avro files (~30kb each). This does not seem right at all... but I cannot seem to find any settings that I may have changed to go from my previous output of 1 large avro to 845 small avros (except adding another data source).

Would this change anything? And how can I get it back to one or two files??

Thanks!

Answer 1

A possibility is to change your block size. If you want to go back to less files, you can also try to use parquet. Transform your .avro files through a pig script and store it like a .parquet file this will reduce your 845 to less files.

But it isn't necessary to get back to less files except for a performance advantage.

Answer 2

The number of files written by MR job is defined by the number of reducers ran. You can use PARALLEL in Pig script to control the number of reducers.

If you are sure that the final data is small enough (comparable to your block size), you can add PARALLEL 1 to your JOIN statement to make sure that JOIN is translated to 1 reducers and thus writes output in only 1 file.

Answer 3

I solved that using SET pig.maxCombinedSplitSize 134217728;

with SET default_parallel 10; it may still output many small files depending on the PIG job.