尝试以myuser身份运行作业时，权限被拒绝，但错误显示root用户没有足够的权限

Question

I'm trying to get Hadoop MapReduce working, but keep facing access problems. I'm not sure what the problem is, whether it is configuration or something else.

When I'm running, for example, this wordcount

hadoop jar /usr/share/hadoop/hadoop-examples-1.2.1.jar wordcount /user/myuser/input.txt /user/myuser/output.txt

I'm getting the following error:

14/09/10 20:15:51 INFO input.FileInputFormat: Total input paths to process : 1
14/09/10 20:15:51 INFO mapred.JobClient: Running job: job_201409101946_0010
14/09/10 20:15:52 INFO mapred.JobClient:  map 0% reduce 0%
14/09/10 20:15:52 INFO mapred.JobClient: Task Id : attempt_201409101946_0010_m_000002_0,         Status : FAILED
Error initializing attempt_201409101946_0010_m_000002_0:
org.apache.hadoop.security.AccessControlException:         
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, 
access=EXECUTE, inode="job_201409101946_0010":hadoop:supergroup:rwx------
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
...

Obviously, user=root cannot access directory owned by the user hadoop. But the problem is that I'm running the job as myuser and I'm not sure why root is involved here at all. Do you know what can be causing this issue?

Answer 1

First of all, /user/myuser/ should be a path on HDFS, not a local path. Then, it needs to exist, so, if it doesn't, run:

hadoop dfs -mkdir /user/myuser/ and then
hadoop dfs -chown myuser:groopOfMyuser /user/myuser/ , where groupOfMyuser is the group to which myuser belongs.

To check whether it exists run: hadoop dfs -ls /user/

Then, to upload your files to the input dir on HDFS, use the command:

hadoop dfs -copyFromLocal /local/path/input.txt /user/myuser/

Note that input and output paths should be directories (on HDFS) and not files. So, the correct command to run your programm would be:

hadoop jar /usr/share/hadoop/hadoop-examples-1.2.1.jar wordcount /user/myuser /user/myuser/output

Finally, check if the hadoop dirs ( $HADOOP_HOME ) are owned by myuser or hadoop or whatever and run the jars as the one who owns them, or change their ownership with chown .

(Assuming you use the old API, but you can easily find the equivalent commands in the new API, as well.)