Hadoop减少作业成功完成但进程仍然存在并占用内存

Question

There is a 4 node hadoop cluster. One of the main node have a strange bug that is the reduce processes finished the job, but it did not terminate. It makes the memory exceed quickly.

I don't know what cause this problem but it only occurs on one node, others are fine. There is also no any error report of the job, just a processes doing nothing.

Answer 1

I found the problem was. I used a library that would open multi-thread when heavy load. If the threads never stop, the processes remains running either.