如何将外部库添加到Hadoop映射减少任务

Question

I have MyClass.java to define the map-reduce task. MyClass.java contains the definition of mapper, reducer and main. It works properly, but if I try to use/add an external jar, I have the message ClassNotFoundException.

To compile I use the command:

javac -classpath hadoop_library_path:my_library_path -sourcepath code_path/ -d class_path/ path/MyClass.java

I create the jar, and then I run the task:

hadoop jar maclass.jar MyClass input output -target target

The external jar need to be added also in in "jar hadoop" command? I tried with the -libjars option with no result. Any idea?

Answer 1

As I commented, I see two options (there could be more):

1. Use Eclipse and generate a runnable jar (I am not sure about NetBeans or IntelliJ).

2. Use maven and its shade plugin to generate an uber jar. You should add all the external libraries that you use as dependencies.

I recommend the latter option.