簡體   English   中英

如何獲得hadoop wordcount示例工作?

[英]how to get hadoop wordcount example working?

大家好,今天我准備運行時,按照此教程http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/設置了hadoop多節點集群它給我的示例程序

 11/07/31 13:44:30 INFO input.FileInputFormat: Total input paths to process : 4
11/07/31 13:44:31 INFO mapred.JobClient: Running job: job_201107311331_0001
11/07/31 13:44:32 INFO mapred.JobClient:  map 0% reduce 0%
11/07/31 13:44:55 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_0, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:44:56 INFO mapred.JobClient:  map 25% reduce 0%
11/07/31 13:44:58 INFO mapred.JobClient:  map 50% reduce 0%
11/07/31 13:45:10 INFO mapred.JobClient:  map 75% reduce 8%
11/07/31 13:45:22 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_1, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:45:22 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:22 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:29 INFO mapred.JobClient: Task Id : attempt_201107311331_0001_m_000002_2, Status : FAILED
java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1590)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1581)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:423)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:521)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:762)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)

11/07/31 13:45:29 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:29 WARN mapred.JobClient: Error reading task outputACRS-HCN1
11/07/31 13:45:44 INFO mapred.JobClient: Job complete: job_201107311331_0001
11/07/31 13:45:44 INFO mapred.JobClient: Counters: 19
11/07/31 13:45:44 INFO mapred.JobClient:   Job Counters
11/07/31 13:45:44 INFO mapred.JobClient:     Launched reduce tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=94857
11/07/31 13:45:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/07/31 13:45:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/07/31 13:45:44 INFO mapred.JobClient:     Launched map tasks=7
11/07/31 13:45:44 INFO mapred.JobClient:     Data-local map tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=45085
11/07/31 13:45:44 INFO mapred.JobClient:     Failed map tasks=1
11/07/31 13:45:44 INFO mapred.JobClient:   FileSystemCounters
11/07/31 13:45:44 INFO mapred.JobClient:     HDFS_BYTES_READ=662018
11/07/31 13:45:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=330575
11/07/31 13:45:44 INFO mapred.JobClient:   File Input Format Counters
11/07/31 13:45:44 INFO mapred.JobClient:     Bytes Read=661664
11/07/31 13:45:44 INFO mapred.JobClient:   Map-Reduce Framework
11/07/31 13:45:44 INFO mapred.JobClient:     Map output materialized bytes=267146
11/07/31 13:45:44 INFO mapred.JobClient:     Combine output records=18043
11/07/31 13:45:44 INFO mapred.JobClient:     Map input records=12761
11/07/31 13:45:44 INFO mapred.JobClient:     Spilled Records=18043
11/07/31 13:45:44 INFO mapred.JobClient:     Map output bytes=1086767
11/07/31 13:45:44 INFO mapred.JobClient:     Combine input records=109848
11/07/31 13:45:44 INFO mapred.JobClient:     Map output records=109848
11/07/31 13:45:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=354

有人可以告訴我出什么事了嗎

java.io.FileNotFoundException: File does not exist: /user/hduser/gutenberg/gutenberg

好的,每個人我都知道問題出在哪里,我必須在其中加*

bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output

這就是教程中應該的樣子

bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg/* /user/hduse/gutenberg-output3

暫無
暫無

聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM