简体   繁体   中英

Error when execute Map-Reduce program

I use hadoop3.1.0 to run Mapreduce WordCount program on Ubuntu,but it always got this INFO.

I saw someone ask this similar question before,but that can't work.

I want to know which file should i modify,or something that i miss.

My java program is from here .

master@kevin-VirtualBox:~/MapReduceTutorial$ $HADOOP_HOME/bin/hadoop jar ProductSalePerCountry.jar /inputMapReduce /mapreduce_output_sales

$HADOOP_HOME/bin/hadoop jar ProductSalePerCountry.jar /inputMapReduce /mapreduce_output_sales
2018-05-20 00:58:37,856 INFO client.RMProxy: Connecting to ResourceManager at kevin-VirtualBox/127.0.1.1:8032
2018-05-20 00:58:38,511 INFO client.RMProxy: Connecting to ResourceManager at kevin-VirtualBox/127.0.1.1:8032
2018-05-20 00:58:38,980 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2018-05-20 00:58:39,058 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/master/.staging/job_1526748071526_0004
2018-05-20 00:58:39,579 INFO mapred.FileInputFormat: Total input files to process : 1
2018-05-20 00:58:39,773 INFO mapreduce.JobSubmitter: number of splits:2
2018-05-20 00:58:39,926 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-05-20 00:58:40,251 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1526748071526_0004
2018-05-20 00:58:40,254 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-05-20 00:58:40,742 INFO conf.Configuration: resource-types.xml not found
2018-05-20 00:58:40,744 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-05-20 00:58:40,930 INFO impl.YarnClientImpl: Submitted application application_1526748071526_0004
2018-05-20 00:58:41,095 INFO mapreduce.Job: The url to track the job: http://kevin-VirtualBox:8088/proxy/application_1526748071526_0004/
2018-05-20 00:58:41,097 INFO mapreduce.Job: Running job: job_1526748071526_0004

core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>Parent directory for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS </name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. </description>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>/home/master/hdfs/name</value>
</property>
<property>
   <name>dfs.datanode.data.dir</name>
   <value>/home/master/hdfs/data</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>1</value>
</property>
<property>
   <name>dfs.permissions</name>
   <value>true</value>
</property>
</configuration>

yarn-site.xml

   <configuration>
        <property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
         </property>
        <property>
         <name>yarn.resourcemanager.hostname</name>
         <value>kevin-VirtualBox</value>
          </property>
    </configuration>

mapred-site.sml

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
<property>  
 <name>mapreduce.application.classpath</name>  
</configuration>

jps

4948 Jps
2856 NodeManager
2088 NameNode
2731 ResourceManager
2207 DataNode

My URL to track the job

在此处输入图片说明

在此处输入图片说明

在此处输入图片说明

Thanks in advance

Thanks @cricket_007 My problem is i don't give memmory to YARN

sets the maximum memory YARN can utilize In yarn-site.xml

<name>yarn.nodemanager.resource.memory-mb</name>
<value>40960</value>

specifying the minimum unit of RAM to allocate

<name>yarn.scheduler.minimum-allocation-mb</name>
 <value>2048</value>

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM