在Giraph上做自己的工作

Question

So, I've successfully executed the SimpleShortestPathComputation on my computer via the script shown here:

#VARIABLES
user_dir=/user/hduser
jar=giraph-examples/target/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar
runner=org.apache.giraph.GiraphRunner
computation=org.apache.giraph.examples.SimpleShortestPathsComputation
informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat

#DELETE PREVIOUS
bin/hdfs dfs -rm -r $user_dir/output/shortestpaths

#GIRAPH JOB
bin/hadoop jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000 $computation -vif $informat -vip $user_dir/input/tiny_graph.txt -vof $outformat -op $user_dir/output/shortestpaths -w 1

Now the problem is, I'm trying to run my own job. It's actually a direct copy-paste from the SimpleShortestPathComputation class, only changed the package name and class name. Trying to run it with -libjars. Here's the full script:

#VARIABLES
user_dir=/user/hduser
jar=giraph-examples/target/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar
runner=org.apache.giraph.GiraphRunner
computation=org.apache.giraph.examples.SimpleShortestPathsComputation
informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat
libjars=/usr/local/hadoop-2.4.0/lib/giraphtrials.jar,$GIRAPH_HOME/giraph-core.jar

#Setup class paths
export HADOOP_CLASSPATH=/usr/local/hadoop-2.4.0/lib/giraphtrials.jar:$GIRAPH_HOME/$jar:$HADOOP_CLASSPATH

#DELETE PREVIOUS
bin/hdfs dfs -rm -r $user_dir/output/shortestpaths

#GIRAPH JOB
bin/hadoop jar $GIRAPH_HOME/$jar $runner -libjars $libjars \
GiraphAlgs.GiraphPBFS -vif $informat -vip $user_dir/input/tiny_graph.txt \
-vof $outformat -op $user_dir/output/shortestpaths -w 1

As you can see, I've tried to use -libjars and HADOOP_CLASSPATH suggestion from this Stackoverflow question to make it work but unfortunately it still gives me a ClassNotFoundException. For better or worse, it doesn't throw it at me in the terminal anymore (it used to): terminal picture . As you can see, it only fails with a general container message now.

Unfortunately, it still gives me the Java.lang.ClassNotFoundException in the logs: log picture . Using Hadoop 2.4.0 and Giraph 1.1.0. I'm running out of ideas what might be wrong with my Giraph and starting to think if I should change careers.

Answer 1

You need to have the jar which contains the class GiraphAlgs.GiraphPBFS in the hadoop classpath.

Also, verify that your classpath is correct set by running $bin/hadoop classpath .

Once in hadoop 2.7 setting HADOOP_CLASSPATH variable didn't work, I had to copy the jar in the hadoop share lib directory: HADOOP_HOME/share/hadoop/mapreduce/lib .