[英]SPARK YARN: cannot send job from client (org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032)
I'm trying to send spark job to yarn (without HDFS) in HA mode. 我正在尝试在HA模式下将火花作业发送到纱线(没有HDFS)。
For submitting I'm using org.apache.spark.deploy.SparkSubmit
. 对于提交,我使用的是
org.apache.spark.deploy.SparkSubmit
。 When I send request from machine with active Resource Manager, it works well. 当我从具有活动资源管理器的计算机发送请求时,它运行良好。 But if I' trying to send from machine with standby Resource Manager, job fails with error:
但是,如果我尝试使用备用资源管理器从计算机发送邮件,则作业将失败并显示以下错误:
DEBUG org.apache.hadoop.ipc.Client - Connecting to spark2-node-dev/10.10.10.167:8032
DEBUG org.apache.hadoop.ipc.Client - Connecting to /0.0.0.0:8032
org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep
However, when I send request via command line (spark-submit), it works well through both active and standby machine. 但是,当我通过命令行(spark-submit)发送请求时,它在主用计算机和备用计算机上都能正常工作。
What can cause the problem? 是什么原因引起的?
PS Use the same parameters for both type of sending job: org.apache.spark.deploy.SparkSubmit
and spark-submit
command line request. PS对两种类型的发送作业使用相同的参数:
org.apache.spark.deploy.SparkSubmit
和spark-submit
命令行请求。 And properties yarn.resourcemanager.hostname.rm_id
defined for all rm hosts 并为所有rm主机定义了属性
yarn.resourcemanager.hostname.rm_id
The problem was with absence of yarn-site.xml within class path for spark-submitter jar. 问题是在火花发送器jar的类路径中缺少yarn-site.xml。 Actually spark submitter jar does not take to account
YARN_CONF_DIR
or HADOOP_CONF_DIR
env var, so cannot see yarn-site. 实际上,火花提交器jar并没有考虑
YARN_CONF_DIR
或HADOOP_CONF_DIR
YARN_CONF_DIR
,因此看不到纱线位置。
One solution that I found was to put yarn-site into classpath of jar. 我发现的一种解决方案是将纱线位置放入jar的类路径中。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.