当输入文件小于地图节点时，hadoop调度程序如何工作

Question

In my situation I need to know if I have 2 job in Jobcontrol and then I have 200 map node my block size is 64 MB so full processing data should be (64*200 = 12.8GB) but firsr job size is 10 GB how hadoop do if map node has empty slots ? Hadoop will be process the second job in queue or waiting until first job map reduce finish and process second job or another way please suggest

Answer 1

If you're using Hadoop's FIFO scheduler and running 2 jobs as same user then, the second job will start running only when there are enough free Map/Reduce slots available while running the first job. Even if there are any, preference is given by scheduler to the first job over second. So, Hadoop will wait for the first job to finish enough before the second job is scheduled.

Usually it is not suggested to have files smaller than the input split size, because the namenode has to manage lot more file inodes when compared to a single large file.