在 Hadoop 中，如果 mapper 或 reducer 清理方法失败会发生什么

Question

Referring to this method: https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/mapreduce/Mapper.html#cleanup(org.apache.hadoop.mapreduce.Mapper.Context)

• If the mapper/reducer cleanup method fails, does the mapper/reducer task fail?
• If the mapper/reducer cleanup method fails, will Hadoop retry the entire task?

Thanks

Answer 1

Applications can override the cleanup(Context) method to perform any required cleanup.

It shall be called per Mapper task or Reducer task. The cleanup method shall be called at the end of each task. If there are N mappers in execution, the corresponding cleanup will be called N times and in case if you are running M reducers in the code, it shall be invoked M times.

cleanup can be used to free the allocated memory or delete the temporary file/directory or other processing as per requirement(Like, context.write(key, value) can be used to emit a value from the cleanup method of mapper and it will be sent to reducers)

If the mapper/reducer cleanup method fails, does the mapper/reducer task fail?

If the mapper/reducer cleanup method fails, will Hadoop retry the entire task?

If task could not cleanup (in exception block), a separate task will be launched with same attempt-id to do the cleanup.

Refer to this link for more information : https://hadoop.apache.org/docs/r2.8.5/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html