使用PIG处理小文件

Question

According to my understanding Map/Reduce works better with large files. ( I understand its due to splitting logic ,etc ), we can put files as values and file name as key in the sequence files and optimize.

Now the issue is I am using PIG for analytics, and we have around thousands of files but all are in KB. As we know pig latin is converted and run as MR jobs, so I've a doubt that MR jobs will be in-efficient owing to small files.

Is there any way by which I can get some control over small files handling over pig ? Is there any out of the box solution?

Answer 1

Pig具有将小文件组合成更大块的功能： http ： //pig.apache.org/docs/r0.11.1/perf.html#combine-files