[英]Why would Oozie fail a job with Error Code LimitExceededException when yarn reports that oozie launcher & mapreduce job have completed successfully?
There are a few questions similar to this on SO. SO上有一些与此类似的问题。 However nothing has worked for me.然而,没有什么对我有用。 So I am posting this question.所以我发布了这个问题。
I am Using CDH 6.2.1我正在使用 CDH 6.2.1
I have a workflow that has map-reduce action.我有一个具有 map-reduce 操作的工作流。 The map-reduce job creates a lot of counters (I think m/r job produces ~300 counters). map-reduce 作业会创建很多计数器(我认为 m/r 作业会产生约 300 个计数器)。
I have set the cdh/yarn/config mapreduce.job.counters.max
property to 8192.我已将 cdh/yarn/config mapreduce.job.counters.max
属性设置为 8192。
I have also set the:我还设置了:
YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml
YARN Service MapReduce Advanced Configuration Snippet (Safety Valve)
MapReduce Client Advanced Configuration Snippet (Safety Valve) for mapred-site.xml
If I run the map-reduce job as a stand-alone yarn job (using yarn jar
command on the command-line), the job completes successfully.如果我将 map-reduce 作业作为独立的 yarn 作业运行(在命令行上使用yarn jar
命令),则作业成功完成。
When I run the job as part of the workflow:当我将作业作为工作流程的一部分运行时:
LimitExceededException: Too many counters: 121 max=120
然而 oozie 未能完成作业报告: LimitExceededException: Too many counters: 121 max=120
The configuration for the mapreduce job & oozie launcher as reported by yarn has the setting: yarn 报告的 mapreduce 作业和 oozie 启动器的配置具有以下设置:
<property>
<name>mapreduce.job.counters.max</name>
<value>8192</value>
<final>true</final>
<source>yarn-site.xml</source>
</property>
Oozie web interface System-Info/OS-Env reports that the following HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/459-oozie-OOZIE_SERVER/yarn-conf/ Oozie web 接口 System-Info/OS-Env 报告以下 HADOOP_CONF_DIR:/var/run/cloudera-scm-agent/process/459-oozie-OOZIE_SERVER/yarn-conf/
In that folder I can see that the mapred-site.xml also has:在那个文件夹中,我可以看到 mapred-site.xml 也有:
<!--'mapreduce.job.counters.max', originally set to '8192' (final), is overridden below by a safety valve-->
<property>
<name>mapreduce.job.counters.max</name>
<value>8192</value>
<final>true</final>
</property>
However I cannot find that property in the yarn-site.xml.但是我在 yarn-site.xml 中找不到该属性。
I am not sure what else I can do at this point...我不确定此时我还能做些什么......
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.