[英]error while executing python mapreduce tasks in hadoop?
I have written mapper
and reducer
for the wordcount
example in python. 我已经为python中的
wordcount
示例编写了mapper
和reducer
。 The scripts works fine as a standalone ones. 这些脚本可以作为独立脚本正常工作。 but I get error when run in hadoop.
但是在hadoop中运行时出现错误。
I am using hadoop2.2
我正在使用
hadoop2.2
Here is my command: 这是我的命令:
hadoop jar share/hadoop/tools/sources/hadoop-streaming*.jar -mapper wordmapper.py -reducer wordreducer.py -file wordmapper.py -file wordreducer.py -input /data -output/output/result7
Exception in thread "main" java.lang.ClassNotFoundException: share.hadoop.tools.sources.hadoop-streaming-2.2.0-test-sources.jar
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:249)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
how to fix this? 如何解决这个问题?
Can u please try it with 你可以试试看吗
hadoop jar $HADOOP_PREFIX/hadoop/tools/sources/hadoop-streaming*.jar -mapper 'wordmapper.py' -reducer 'wordreducer.py' -file $CODE_FOLDER/wordmapper.py -file $CODE_FOLDER/wordreducer.py -input /data -output /output/result7 hadoop jar $ HADOOP_PREFIX / hadoop / tools / sources / hadoop-streaming * .jar -mapper'wordmapper.py'-reducer'wordreducer.py'-文件$ CODE_FOLDER / wordmapper.py -file $ CODE_FOLDER / wordreducer.py -input /数据输出/输出/结果7
Where $HADOOP_PREFIX is folder location where the hadoop is placed on your machine. $ HADOOP_PREFIX是在您的机器上放置Hadoop的文件夹位置。 for eg./usr/local/ for my machine.
用于我的机器的例如/ usr / local /。 If you can manually acces that location and check whether that jar is present.
如果可以手动访问该位置,然后检查该jar是否存在。
And $CODE_FOLDER contains the code file where the script is saved. $ CODE_FOLDER包含用于保存脚本的代码文件。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.