Hadoop上的外部/ jar文件存储在哪里？

Question

Lets say I write a WordCount example, and then in the eclipse project include an external jar file such as MyJar.jar. Now if I export the whole WordCount project as a word.jar file, and then type

$> hadoop jar word.jar WordCount input output

I understand that the job executes and the word.jar will have a lib directory that contains MyJar.jar file. Now, where on the HDFS will this jar file MyJar file be stored when the job is running that makes calls to methods of this jar file?

Answer 1

The bin/hadoop script actually unpacks your work.jar file into a tmp folder on the local file system.

The Job client handles the creation of a job folder in HDFS where your original jar, all the lib jars and other job files (such as the job.xml, distributed cache files etc) are uploaded to.

When your job runs on a cluster node, these files are copied back down to a tmp job directory on the local file system of that node. For efficiency reasons the files are only copied down once, rather than for each map tasks which runs on that node.