stackoom
  简体   繁体   中英

Why does hadoop in Windows7 raised a RuntimeException: Error in configuring object

 原文  2016-03-25 12:10:16  4   1    python/ hadoop/ windows-7/ hadoop-streaming/ runtimeexception

Question

I'm trying to run a hadoop program in pseudo-distributed mode in windows7, and a RuntimeException has been raised. I don't know why and how to fix it.

I use python for mapreduce work instead of java using hadoop-streaming utility( https://hadoop.apache.org/docs/current/hadoop-streaming/HadoopStreaming.html ). I ran hadoop jar %HADOOP_HOME%\\share\\hadoop\\tools\\lib\\hadoop-streaming-2.6.2.jar -mapper mapper.py -reducer reducer.py -input myinput -output myoutput -file mapper.py -file reducer.py and a java.lang.RuntimeException: Error in configuring object has been raised and job failed.

My hadoop version is 2.6.2, and hadoop is built on windows7 x64, not on virtual machine or cygwin.

First of all, I used the standard test program of hadoop to test whether hadoop is installed successfully. I ran yarn jar %HADOOP_HOME%/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar pi 16 1000 and the output is correct. Hadoop and MapReduce work.

Secondly, the mapper.py and reducer.py have been tested under command prompt using command cat data.txt | python mapper.py | sort | python reducer.py cat data.txt | python mapper.py | sort | python reducer.py cat data.txt | python mapper.py | sort | python reducer.py and the output is correct. Thus, it wouldn't be the python program's bug.

I try to change the 'hadoop.tmp.dir' property to local directory and /tmp, but neither worked. I don't know why the exception was raised and how to fix it. Thanks a lot if anyone can help me!

The complete Exception log is as follows: 

16/03/25 18:47:46 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [mapper.py, reducer.py, /C:/Users/chensk/AppData/Local/Temp/hadoop-unjar5079915111177028733/] [] C:\Users\chensk\AppData\Local\Temp\streamjob6081649198401076264.jar tmpDir=null
16/03/25 18:47:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/03/25 18:47:48 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/03/25 18:47:50 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/25 18:47:50 INFO mapreduce.JobSubmitter: number of splits:2
16/03/25 18:47:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1458902651843_0002
16/03/25 18:47:51 INFO impl.YarnClientImpl: Submitted application application_1458902651843_0002
16/03/25 18:47:51 INFO mapreduce.Job: The url to track the job: http://skyfalls:8088/proxy/application_1458902651843_0002/
16/03/25 18:47:51 INFO mapreduce.Job: Running job: job_1458902651843_0002
16/03/25 18:47:59 INFO mapreduce.Job: Job job_1458902651843_0002 running in uber mode : false
16/03/25 18:47:59 INFO mapreduce.Job:  map 0% reduce 0%
16/03/25 18:48:07 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000003\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:08 INFO mapreduce.Job:  map 50% reduce 0%
16/03/25 18:48:08 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000002\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:09 INFO mapreduce.Job:  map 0% reduce 0%
16/03/25 18:48:14 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000001_1, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000004\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:16 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000005\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:23 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000006\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:25 INFO mapreduce.Job: Task Id : attempt_1458902651843_0002_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "G:\Programs\hadoop\tmp\nm-local-dir\usercache\chensk\appcache\application_1458902651843_0002\container_1458902651843_0002_01_000007\.\mapper.py": CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 23 more
Caused by: java.io.IOException: CreateProcess error=193, %1 ??????完?? Win32 ??車
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

16/03/25 18:48:32 INFO mapreduce.Job:  map 100% reduce 100%
16/03/25 18:48:34 INFO mapreduce.Job: Job job_1458902651843_0002 failed with state FAILED due to: Task failed task_1458902651843_0002_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/03/25 18:48:34 INFO mapreduce.Job: Counters: 13
    Job Counters 
        Failed map tasks=7
        Killed map tasks=1
        Launched map tasks=8
        Other local map tasks=6
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=52274
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=52274
        Total vcore-seconds taken by all map tasks=52274
        Total megabyte-seconds taken by all map tasks=53528576
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
16/03/25 18:48:34 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!

1 answers

solution1
 0 ACCPTED  2016-03-25 13:33:59

The values of the -mapper and -reducer command-line arguments need be executables. On Windows, which doesn't support shebang lines, Python scripts aren't executable.

Try specifying the -mapper and -reducer arguments as follows: 

-mapper "python mapper.py" -reducer "python reducer.py"

Please note that I haven't run this myself so I can't guarantee that it will work. It may complain that it can't find a file whose name is python mapper.py .

If this doesn't work, write short batch scripts mapper.bat and reducer.bat that just run the corresponding Python script. For example, mapper.bat would contain the following lines: 

@echo off
python mapper.py %*

Then, specify these batch scripts as your -mapper and -reducer arguments. Also, add the command-line arguments -file mapper.bat and -file reducer.bat to ensure that these batch scripts get copied to all your compute nodes.

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Hadoop: Error: java.lang.RuntimeException: Error in configuring object Hadoop Mapreduce Job failes :java.lang.RuntimeException: Error in configuring object Why does 'raised Error' work, but 'assert' doesn't? Pyside: Unresolved Functions error messages on windows7 When configuring Geodjango, gdal1.10 and python3.3 are incompatible on Windows7 python 2.7.9/pycharm 4/windows7: 'file' object has no attribute 'readall' Error Installing Tornado 4.0 Windows7 64bit SparkContext not found on windows7 installing Scrapy under Windows7 why sublime text3 and windows7 notepad have difference in displaying the same *.txt file?
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM