Hadoop：为什么编写一个精简的精简任务会超时？

Question

I have a Hadoop reduce task that reads its input records in batches and does a lot of processing and writes a lot of output for each input batch. I have read that Hadoop considers writing output to be "progress" for the purpose of killing hung tasks. However, despite constantly writing lots of output, my task is still being timed out and killed. So: how can I find out when Hadoop thinks a task last reported progress? Why would I have to call context.progress() with every context.write() ? Are there any situations where writing is not counted as progress? (For instance, my keys are NullWritable s.) I'm using Cloudera CDH3u1 on CentOS 5.7 if that makes any difference.

Answer 1

Not sure why the tasks are getting killed, but you could increase the value mapreduce.task.timeout , it's defaulted to 600000 ms. This might not be a good practice as rouge tasks will run for more time because of the increase in the timeout value.