在Eclipse中调试Hadoop

Question

Is it possible to debug Hadoop's source code in Eclipse?I'm not asking about the map reduce tasks. I want to see which part of the Hadoop source code is responsible for scheduling the map reduce tasks and how it works. Is there any mechanism by which it can be done?

Answer 1

You can download Hadoop project and integrate it to your eclipse, and use F5 or F6 to debug. You have different mode of debugging in eclipse:

1. F5 : Step by Step debugging
2. F6 : Skips loops and Subroutines
3. F7 : Skips the loop or subroutine and returns to the last cursor point.
4. F8 : Execute and come out of debugging

Or you can try yourself to understand the workflow by following step by step, you can begin from your run() method in your main.

To answer your question: who does schedule the map task?

As you can see in this schema, files are divided by the InputFormat class into fixed-size pieces called InputSplits. Each split is then given to a mapper, which is a node that was assigned a map task.

The same InputFormat class also provides a RecordReader responsible for parsing the split and extracting records.Each record is passed to a map function as a (key, value) pair. So the Mapper class is the one who call map methods.

Here is the workflow of the wordcount example:

Where the FileInputFormat is an abstract class that extends the abstract class InputFormat , and the TextInputFormat extends the FileInputFormat class.

Answer 2

Here are instructions from Apache Hadoop documentation. I haven't tried them out, but the instructions are good enough to get started.