简体   繁体   English

在 Hadoop 上运行 MapReduce 字数会给出异常消息:系统找不到指定的路径

[英]Running MapReduce word count on Hadoop gives Exception message: The system cannot find the path specified

this is my first Stack Overflow question ever.这是我有史以来的第一个 Stack Overflow 问题。 I've setup my hadoop (2.9.2) single node cluster in pseudo distributed mode.我已经在伪分布式模式下设置了我的 hadoop (2.9.2) 单节点集群。 When i try to run hadoop jar C:/MapReduceClient.jar wordcount /input_dir /output_dir, i get the following log with error当我尝试运行 hadoop jar C:/MapReduceClient.jar wordcount /input_dir /output_dir 时,我收到以下错误日志

19/01/16 20:19:17 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/01/16 20:19:18 INFO input.FileInputFormat: Total input files to process : 1
19/01/16 20:19:19 INFO mapreduce.JobSubmitter: number of splits:1
19/01/16 20:19:19 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/01/16 20:19:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547662294790_0002
19/01/16 20:19:19 INFO impl.YarnClientImpl: Submitted application application_1547662294790_0002
19/01/16 20:19:19 INFO mapreduce.Job: The url to track the job: http://DESKTOP-XXXXXX:8088/proxy/application_1547662294790_0002/
19/01/16 20:19:19 INFO mapreduce.Job: Running job: job_1547662294790_0002
19/01/16 20:19:19 INFO mapreduce.Job: Running job: job_1547662294790_0002
19/01/16 20:19:27 INFO mapreduce.Job: Job job_1547662294790_0002 running in uber mode : false
**19/01/16 20:19:27 INFO mapreduce.Job:  map 0% reduce 0%**
**19/01/16 20:19:27 INFO mapreduce.Job: Job job_1547662294790_0002 failed with state FAILED due to: Application application_1547662294790_0002 failed 2 times due to AM Container for appattempt_1547662294790_0002_000002 exited with  exitCode: 1**
Failing this attempt.Diagnostics: [2019-01-16 20:19:25.234]Exception from container-launch.
Container id: container_1547662294790_0002_02_000001
Exit code: 1
**Exception message: The system cannot find the path specified.**
The system cannot find the path specified.
The system cannot find the path specified.


[2019-01-16 20:19:25.236]Container exited with a non-zero exit code 1.
[2019-01-16 20:19:25.236]Container exited with a non-zero exit code 1.
For more detailed output, check the application tracking page: http://DESKTOP-XXXXX:8088/cluster/app/application_1547662294790_0002 Then click on links to logs of each attempt
. Failing the application.
19/01/16 20:19:28 INFO mapreduce.Job: Counters: 0

The same setup with the same .jar is working on my other pc and the output is correct.具有相同 .jar 的相同设置正在我的另一台电脑上运行,并且输出是正确的。 Windows 10 Pro x64 (both) Only difference is that the working one has java 1.8.0_171 installed Windows 10 Pro x64(两者)唯一的区别是工作的一个安装了 java 1.8.0_171

JAVA_HOME= C:\\Java\\jdk1.8.0_201 HADOOP_HOME= C:\\hadoop-2.9.2 JAVA_HOME= C:\\Java\\jdk1.8.0_201 HADOOP_HOME= C:\\hadoop-2.9.2

PATH=%JAVA_HOME%\\bin;C:\\hadoop-2.9.2\\bin PATH=%JAVA_HOME%\\bin;C:\\hadoop-2.9.2\\bin

My config files: /etc/hadoop/core-site.xml我的配置文件:/etc/hadoop/core-site.xml

<configuration>
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

/etc/hadoop/mapred-site.xml /etc/hadoop/mapred-site.xml

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

/etc/hadoop/hdfs-site.xml /etc/hadoop/hdfs-site.xml

<configuration>
   <property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
   <property>
       <name>dfs.namenode.name.dir</name>
       <value>C:\hadoop-2.9.2\data\namenode</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
       <value>C:\hadoop-2.9.2\data\datanode</value>
   </property>
</configuration>

/etc/hadoop/yarn-site.xml /etc/hadoop/yarn-site.xml

<configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
   </property>
   <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

Solved.解决了。 It was the default user name being non latin characters messing the node manager up.默认的用户名是非拉丁字符,弄乱了节点管理器。 Checked with a whoami command only to find out that the default user name was "???????"使用 whoami 命令检查后才发现默认用户名是“???????”

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM