使用Yarn-client在Google Cloud上的Hadoop中运行JAR

Question

我想使用Yarn-client在Google Cloud上的Hadoop中运行JAR。

我在hadoop的主节点中使用此命令

spark-submit --class find --master yarn-client find.jar

但它返回此错误

    15/06/17 10:11:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop-m-on8g/10.240.180.15:8032
15/06/17 10:11:07 INFO ipc.Client: Retrying connect to server: hadoop-m-on8g/10.240.180.15:8032. Already tried 0 
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

问题是什么？ 如果有用，这是我的yarn-site.xml

<?xml version="1.0" ?>
<!--
     <configuration>
      <!-- Site specific YARN configuration properties -->
      <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/yarn-logs/</value>
        <description>
          The remote path, on the default FS, to store logs.
        </description>
      </property>
      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>
      <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-m-on8g</value>
      </property>
      <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>5999</value>
        <description>

Answer 1

就您而言，由于未知原因，YARN ResourceManager可能不正常； 您可以尝试使用以下方法固定纱线：

sudo sudo -u hadoop /home/hadoop/hadoop-install/sbin/stop-yarn.sh
sudo sudo -u hadoop /home/hadoop/hadoop-install/sbin/start-yarn.sh

但是，您似乎正在使用“即点即用”解决方案。 由于存在一些错误和缺少内存配置，因此，Click-to-Deploy的Spark + Hadoop 2部署目前实际上不支持YARN上的Spark。 如果您只是尝试使用--master yarn-client开箱即用，通常会遇到类似这样的情况：

15/06/17 17:21:08 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: -1
   appStartTime: 1434561664937
   yarnAppState: ACCEPTED

15/06/17 17:21:09 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: -1
   appStartTime: 1434561664937
   yarnAppState: ACCEPTED

15/06/17 17:21:10 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: 0
   appStartTime: 1434561664937
   yarnAppState: RUNNING

15/06/17 17:21:15 ERROR cluster.YarnClientSchedulerBackend: Yarn application already ended: FAILED
15/06/17 17:21:15 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/metrics/json,null}
15/06/17 17:21:15 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}

支持良好部署的方法是在Google Compute Engine上使用Hadoop 2且将Spark配置为可以在YARN上运行的集群是使用bdutil 。 您将运行类似：

./bdutil -P <instance prefix> -p <project id> -b <bucket> -z <zone> -d  \
    -e extensions/spark/spark_on_yarn_env.sh generate_config my_custom_env.sh
./bdutil -e my_custom_env.sh deploy

# Shorthand for logging in to the master
./bdutil -e my_custom_env.sh shell

# Handy way to run a socks proxy to make it easy to access the web UIs
./bdutil -e my_custom_env.sh socksproxy

# When done, delete your cluster
./bdutil -e my_custom_env.sh delete

使用spark_on_yarn_env.sh Spark应该默认为yarn-client ，尽管您可以随时重新指定--master yarn-client 。 您可以使用./bdutil --help来bdutil ./bdutil --help可用标志的更详细说明。 以下是我上面包含的标志的帮助条目：

-b, --bucket
  Google Cloud Storage bucket used in deployment and by the cluster.

-d, --use_attached_pds
  If true, uses additional non-boot volumes, optionally creating them on
  deploy if they don't exist already and deleting them on cluster delete.

-e, --env_var_files
  Comma-separated list of bash files that are sourced to configure the cluster
  and installed software. Files are sourced in order with later files being
  sourced last. bdutil_env.sh is always sourced first. Flag arguments are
  set after all sourced files, but before the evaluate_late_variable_bindings
  method of bdutil_env.sh. see bdutil_env.sh for more information.

-P, --prefix
  Common prefix for cluster nodes.

-p, --project
  The Google Cloud Platform project to use to create the cluster.

-z, --zone
  Specify the Google Compute Engine zone to use.