解决mapreduce.Cluster由于实例化YarnClient中的错误，无法使用org.apache.hadoop.mapred.YarnClientProtocolProvider

Question

我主要根据以下指南在测试的单节点群集上设置了基本的Hadoop 2.6.0 + Hive 0.14.0：

Hadoop： http ： //tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/

蜂巢： https : //cwiki.apache.org/confluence/display/Hive/GettingStarted

但是，如果我运行WordCount java示例，则会出现此错误：

$ hadoop jar wc.jar WordCount /user/hadoop/HDFSClient.java /user/hadoop/wc.out
16/02/08 12:24:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/08 12:24:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/08 12:24:56 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at WordCount.main(WordCount.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

conf / hdfs-site.xml：

<configuration>

<property>
 <name>dfs.replication</name>
  <value>1</value>
  </property>

<property>
  <name>dfs.name.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
      </property>

<property>
  <name>dfs.data.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
      </property>

<property>
  <name>dfs.namenode.http-address</name>
      <value>0.0.0.0:8000</value>
      </property>

</configuration>

conf / yarn-site.xml：

<configuration>

 <property>
   <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce.shuffle</value>
        </property>

 <property>
   <name>yarn.resourcemanager.webapp.address</name>
     <value>0.0.0.0:8004</value>
        </property>
</configuration>

conf / core-site.xml：

<configuration>

<property>
  <name>mapred.job.tracker.http.address</name>
      <value>50031</value>
   </property>
<property>
  <name>fs.default.name</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

我对设置Hadoop和Hive有点陌生，希望对可能要看的内容有任何建议或指点。

Answer 1

将您的core-site.xml，mapred-site.xml，hdfs-site.xml和yarn-site.xml文件属性替换为以下内容

核心的site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

HDFS-site.xml中

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

mapre-site.xml中

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

纱的site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

Answer 2

如果您在类路径中使用“ hadoop-mapreduce-client-jobclient.jar”运行应用程序，这似乎可以正常工作。