简体繁体 English

Mapper或Reducer Task中未处理的异常会使任务失败吗？

[英]An unhandled exception in Mapper or Reducer Task would make the task failed?

原文 2012-07-09 19:22:46 7 2 hadoop/ mapreduce

I am wondering if an unhandled exception in a mapper task (or a reducer task) would make the task failed or only the particular input pair of the time would be ignored? 我想知道映射器任务（或化简器任务）中未处理的异常是否会使任务失败，或者只忽略特定时间的输入对？ I guess it is the former but not so sure. 我猜是前者，但不确定。

2 个解决方案

It depends on the error. 这取决于错误。 Either way, I always make it a point to wrap the map and reduce logic with try-catch blocks. 无论哪种方式，我总是以包装映射和使用try-catch块减少逻辑为重点。 On an error, I increment a counter whose name is derived from the exception class name. 在发生错误时，我增加了一个其名称源自异常类名称的计数器。 This not only protects the rest of hadoop from logic errors, but gives you an idea of how much went wrong just by looking at the logs. 这不仅可以保护hadoop的其余部分免受逻辑错误的影响，而且还可以通过查看日志了解出了什么错误。

If the exception is not recurring (ie a retry will succeed), then it will fail the task but the task will be retried (or a speculative execution running in parallel may proceed). 如果该异常不再发生（即重试将成功），则它将使任务失败，但将重试该任务（或者可以继续并行运行的推测执行）。 If the exception will always occur (eg there is bad input data) then retries will continue to fail and eventually the whole job will fail. 如果总是发生异常（例如，输入数据不正确），则重试将继续失败，最终整个工作将失败。