使用mapreduce处理文件

Question

I use simple pig script that reads the input .txt file and for each line new filed is added.

The output relation is then stored into avro.

Is there any benefit to run such a script in the mapreduce mode compare to local mode?

Thank you

Answer 1

In local mode you are running your job on your local machine. With mapreduce you run your job in a cluster (your file will be splitted into pieces and will be processed on several machines in parallel).

So, in theory, if your file is big enough (or there are lots of files like this to process), you'll be able to accomplish your job in less time with mapreduce mode.