Analyzing Map-Reduce jobs produced by Pig/Hive compiler

Question

Is there anyway to view the code for the Map-Reduce jobs that are produced by both Pig and Hive?

I understand that with Hive, I can view the abstract syntax tree, but it seems that it is not possible to access the actual Java code for the MR jobs. Am I mistaken in that assumption?

Answer 1

Pig and Hive don't generate any Java code, but plan. The plan can be see using the explain command in the shell. One way to generate the Java code from SQL is to use YSmart . Note that there are a lot of changes happening in Hive to make it much faster.

Answer 2

You can download and build the source code yourself.

Then using a java IDE like eclipse you can remote debug you can inspect the code. Although you might not have all dependencies in place and might not be able to inspect all objects you can see the plans in more detail than the Explain function.

To allow remote debugging add the debug parameter to your hadoop bash script:

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1044