使用Virtualbox的Hortonworks Hadoop-执行jar

Question

i am a hadoop beginner. Therefore i try to handle easy examples to get to know the environment. So i created a jar that performs a wordcount - amongst others this one Hadoop: Set up Maven project in 5 Minutes . My problem: It is not possible for me to execute the .jar at my Hortonworks Sandbox. Every time i get the error:

WARNING: Use "yarn jar" to launch YARN applications.
Not a valid JAR: /user/maria_dev/wordcount3.jar

To launch the jar i did the following steps:

1. Create the java classes in eclipse using maven projects.
2. Export a runnable jar file.
3. Load the jar into HDFS (using Ambari).
4. Call the jar using (with root-user, also trying with and without ssh):

hadoop jar /user/maria_dev/wordcount3.jar /user/maria_dev/BlackArrow.txt /user/maria_dev /

AND

hadoop jar /user/maria_dev/wordcount3.jar de.kja. wordcount3. wordcount /user/maria_dev/BlackArrow.txt /user/maria_dev/

In Both cases i get the error from above...

So i had the following thoughts:

1. My java class has an error -> I tried diverse other examples, all do not work. But trying to run the main class of each java file as java application i get one of the following errors:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0 at de.kja.wordcount2.wordcount.main(wordcount.java:47)

OR

Usage: WordCount [generic options] Generic options supported are -conf specify an application configuration file -D use value for given property -fs specify a namenode -jt specify a job tracker -files specify comma separated files to be copied to the map reduce cluster -libjars specify comma separated jar files to include in the classpath. -archives specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions]

2. I read that the jar file does not have to be uploaded to the HDFS. So do have to load it to the local files of the virtual box? If yes: How can i do this?

Hadoop version: 2.7.1.2.4.0.0-169

If you need more information please ask! Thank you for your help :)

Answer 1

In your virtual machine, open the Linux terminal

su - ambari-qa

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen 1000 /tmp/genout

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar terasort /tmp/genout /tmp/sortout

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teravalidate /tmp/sortout /tmp/valout

This is a sample to ensure you able to run jobs from sandbox.

you need to run your code as yarn jar ....