如何在Spark中为纱线客户端模式配置YARN地址？

Question

在使用Spark 1.3的远程scala程序中，如何初始化sparkContext以便可以连接到在YARN上运行的Spark？ 即我应该把YARN节点的地址放在哪里？

目前，我的程序包含：

  val conf = new SparkConf().setMaster("yarn-client").setAppName("MyApp")
  val sc = new SparkContext(conf)

它产生

[error] (run-main-0) java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
    at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
    at SparkExampleLocalDriver$.main(SparkExample.scala:9)
    at SparkExampleLocalDriver.main(SparkExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: org.apache.spark.SparkException: Unable to load YARN support
    at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:217)
    at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:212)
    at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
    at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
    at SparkExampleLocalDriver$.main(SparkExample.scala:9)
    at SparkExampleLocalDriver.main(SparkExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:195)
    at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:213)
    at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:212)
    at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
    at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
    at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
    at SparkExampleLocalDriver$.main(SparkExample.scala:9)
    at SparkExampleLocalDriver.main(SparkExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)

Answer 1

您的Spark二进制文件不包含YARN相关的类

对hadoop使用预构建的二进制文件

http://www.apache.org/dyn/closer.cgi/spark/spark-1.3.1/spark-1.3.1-bin-hadoop2.4.tgz

如果您正在编译源代码，请包括yarn和hadoop配置文件

./make-distribution.sh --tgz --skip-java-test -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0