python mapreduce作业返回错误

Question

Hi I have just started using Hadoop and running my first mapreduce job. I have used python for building the map and reduce scripts and I tested it and its working fine but when I am trying to run them on hadoop.It is returning error.

Following is the command I entered in the terminal

/home/maitreyee/hadoop$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar     -mapper /usr/bin/python mapper1.py -reducer /usr/bin/python reducer1.py -input /user/hduser/gutenberg/* -output /user/hduser/gutenberg-output1

And the below error appears

Warning: $HADOOP_HOME is deprecated.
packageJobJar: [/app/hadoop/tmp/hadoop-unjar3238940252334854546/] []      /tmp/streamjob4553487258055690616.jar tmpDir=null
14/12/05 11:53:29 INFO streaming.StreamJob: Running job: job_201412050953_0004
14/12/05 11:53:29 INFO streaming.StreamJob: To kill this job, run:
14/12/05 11:53:29 INFO streaming.StreamJob: /home/maitreyee/hadoop/libexec/../bin/hadoop job  -Dmapred.job.tracker=localhost:54311 -kill job_201412050953_0004
14/12/05 11:53:29 INFO streaming.StreamJob: Tracking URL: http://localhost:50030   /jobdetails.jsp?jobid=job_201412050953_0004
14/12/05 11:53:30 INFO streaming.StreamJob:  map 0%  reduce 0%
14/12/05 11:54:54 INFO streaming.StreamJob:  map 100%  reduce 100%
14/12/05 11:54:54 INFO streaming.StreamJob: To kill this job, run:
14/12/05 11:54:54 INFO streaming.StreamJob: /home/maitreyee/hadoop/libexec/../bin/hadoop    job  -Dmapred.job.tracker=localhost:54311 -kill job_201412050953_0004
14/12/05 11:54:54 INFO streaming.StreamJob: Tracking URL: http://localhost:50030  /jobdetails.jsp?jobid=job_201412050953_0004
14/12/05 11:54:54 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map          Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask:        task_201412050953_0004_m_000000
14/12/05 11:54:54 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

Kindly suggest what is going wrong and how could it be resolved.

Answer 1

It feels its just a matter of time. Before running jobs in hadoop make sure hadoop is running fine through jps, keep the system updated and check the ssh connection as well. Then write the command as follows to run a simple python map reduce job in hadoop (I am using Ubuntu 12.04 LTS and Hadoop 1.2.1).

 hduser@bharti-desktop:~/hadoop$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar -input /user/hduser/gutenberg/gutenberg/ -output /user/hduser/op4 -mapper /home/hduser/hadoop/mapper1.py -file /home/hduser/hadoop/mapper1.py -reducer /home/hduser/hadoop/reducer1.py -file /home/hduser/hadoop/reducer1.py 

A small explanation of the terminal command above: Since its a streaming job so we first put the location of streaming jar file of hadoop, then the location of the input file, followed by the location of output file (try to give a unique name and should be present in hdfs), then we tell hadoop what function we want to perform, and where it will be performed (map and reduce tasks), followed by the file attribute to tell the location of scripts. (in the case of scripting language.)

If there is still any doubt kindly let me know.