Hadoop distcp作业已成功，但try_xxx被ApplicationMaster终止
[英]Hadoop distcp jobs SUCCEEDED but attempt_xxx killed by ApplicationMaster
问题描述
Running a distcp job I encounter the following problem: Almost all map tasks are marked as successful but with note saying Container killed. 
运行distcp作业时，我遇到以下问题：几乎所有地图任务都被标记为成功，但是注释说Container被杀死。
On the online interface the log for the map jobs says: Progress 100.00 State SUCCEEDED 在在线界面上，地图作业的日志显示：进度100.00状态成功
but under Note it says for almost every attempt (~200) Container killed by the ApplicationMaster. 但是在“注释”下，它表示几乎每一次尝试（〜200）ApplicationMaster杀死的容器。 Container killed by the ApplicationMaster. 容器被ApplicationMaster杀死。 Container killed on request. 集装箱应要求被杀死。 Exit code is 143 退出代码是143
In the log file associated with the attempt I can see a log saying Task 'attempt_xxxxxxxxx_0' done. 在与尝试关联的日志文件中，我可以看到一条日志，说明任务'attempt_xxxxxxxxx_0'已完成。
stderr output is empty for all jobs/attempts. 对于所有作业/尝试，stderr输出为空。
When looking at the application master log and following one of the successful (but killed) attempts I find the following logs: 当查看应用程序主日志并遵循其中一次成功（但已终止）尝试时，我发现以下日志：
2017-01-05 10:27:22,772 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1483370705805_4012_m_000000_0
2017-01-05 10:27:22,773 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1483370705805_4012_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2017-01-05 10:27:22,775 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2017-01-05 10:27:22,775 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1483370705805_4012Job Transitioned from RUNNING to COMMITTING
2017-01-05 10:27:22,776 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2017-01-05 10:27:23,118 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-01-05 10:27:24,125 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e116_1483370705805_4012_01_000002
2017-01-05 10:27:24,126 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2017-01-05 10:27:24,126 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1483370705805_4012_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
i have set "mapreduce.map.speculative=false"! 我已设置“ mapreduce.map.speculative = false”！
All MAP task are SUCCEEDED(distcp job has no REDUCE)，but MAPREDUCE is going for a long time(several hours) , then it is succeeded and distcp job is done. 所有MAP任务均为SUCCEEDED（distcp作业没有REDUCE），但是MAPREDUCE进行了很长时间（几个小时），然后成功完成distcp作业。
I am running 'yarn version'= Hadoop 2.5.0-cdh5.3.1 我正在运行'纱线版本'= Hadoop 2.5.0-cdh5.3.1
Should I be worried about this? 我应该为此担心吗？ And what causes the containers to be killed? 什么原因导致集装箱被杀死？ Any suggestions would be greatly appreciated! 任何建议将不胜感激！
1 个解决方案
解决方案1
0 2017-01-05 09:45:13
Those killed attempts might be due to speculative execution. 那些失败的尝试可能是由于投机执行。 In this case there is nothing to worry about. 在这种情况下，无需担心。
To make sure it is the case, try running your distcp like this: 为了确保是这种情况，请尝试按以下方式运行distcp：
hadoop distcp -Dmapreduce.map.speculative=false ...
You should stop seeing those killed attempts. 您应该停止看到那些被杀死的尝试。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.