为什么使用“java.lang.ClassNotFoundException：org.apache.hadoop.fs.FSDataInputStream”启动带有yarn-client的spark-shell失败？

Question

I am trying to set up a cluster at home for my personal needs (learning). First I made Hadoop+Yarn. MR2 is working. Second - I am trying to add Spark but getting an error about missing classes.

[root@master conf]# spark-shell --master yarn-client
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

I followed these instructions and added into spark-env.sh

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop)

More info :
Cent_OS.86_64
Hadoop dir: /usr/local/hadoop

Hadoop version:

[root@master conf]# hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Manual says that I must have 2 vars: HADOOP_CONF_DIR or YARN_CONF_DIR

[root@master conf]# echo $HADOOP_CONF_DIR
/usr/local/hadoop/etc/hadoop
[root@master conf]# echo $YARN_CONF_DIR
/usr/local/hadoop/etc/hadoop

Spark is spark-1.5.0-bin-without-hadoop.tgz -> /usr/local/spark

I am trying to launch spark-shell --master yarn-client at the same time when hadoop+yarn are up and available http://master:50070/dfshealth.html#tab-overview http://master:8088/cluster/apps http://master:19888/jobhistory

I have no Scala installed if it matters. Any ideas what could I miss in Spark settings? Thank you.

Answer 1

Answering my own question: First of all this is my personal mistake. Calling spark-shell I was launching it from the old (wrong) place /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/bin/spark-shell . I was sure that I've deleted all from CDH testings by yum remove cloudera* .

[root@master bin]# type spark-shell
spark-shell is hashed (/usr/bin/spark-shell)
[root@master bin]# hash -d spark-shell

Now, launching if from old spark-1.5.0-bin-without-hadoop.tgz still gave me the same error. Downloaded spark-1.5.0-bin-hadoop2.6 , added export SPARK_DIST_CLASSPATH=$HADOOP_HOME - spark-shell is working now.

Answer 2

I was getting this error because by typing spark-shell , /usr/bin/spark-shell was getting executed.

To call my specific spark-shell, I ran the following command from inside of own-built spark source -

./bin/spark-shell

Answer 3

Instead of spark-1.5.0-bin-without-hadoop.tgz download one of the builds for Hadoop 2.x. They are simpler to set up as they come with the Hadoop client libraries.