使用Hadoop解析日志文件

Question

Im a newbie to hadoop. I did the setup and executed the basic word count java program. The results look good.

My question is it possible to parse an extremely big log file to fetch only a few required lines using the map/reduce classes? Or is some other step required?

Any pointers in this direction will be very useful. Thanks, Aarthi

Answer 1

Yes it is entirely possible, and if the file is sufficiently large, I believe hadoop could prove to good way to tackle it, despite what nhahtdh says.

Your mappers could simply act as the filters - check the values passed to them, and only if they fit the conditions of a required line do you context.write() it out.

You wont even need to write your own reducer, just use the default reduce() in the Reducer class.