帶hbase錯誤的Apache Nuch 2.2.1

Question

我使用Nutch 2.2.1、4.3.0和HBase 0.90.4 SOLR。

我收到以下錯誤。

InjectorJob: Using class org.apache.gora.hbase.store.HBaseStore as the Gora storage class.
InjectorJob: total number of urls rejected by filters: 0
InjectorJob: total number of urls injected after normalization and filtering: 1
Exception in thread "main" java.lang.RuntimeException: job failed: name=generate: null, jobid=job_local1662982347_0002
    at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54)
    at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199)
    at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:152)
    at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)

在Hadoop日志中是以下內容。

2014-08-11 09:13:43,246 INFO  crawl.InjectorJob - InjectorJob: Using class org.apache.gora.hbase.store.HBaseStore as the Gora storage class.
2014-08-11 09:13:43,293 WARN  util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-08-11 09:13:43,372 WARN  snappy.LoadSnappy - Snappy native library not loaded
2014-08-11 09:13:44,017 INFO  mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
2014-08-11 09:13:44,245 INFO  regex.RegexURLNormalizer - can't find rules for scope 'inject', using default
2014-08-11 09:13:44,381 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2014-08-11 09:13:44,686 INFO  crawl.InjectorJob - InjectorJob: total number of urls rejected by filters: 0
2014-08-11 09:13:44,686 INFO  crawl.InjectorJob - InjectorJob: total number of urls injected after normalization and filtering: 1
2014-08-11 09:13:44,695 INFO  crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2014-08-11 09:13:44,696 INFO  crawl.AbstractFetchSchedule - defaultInterval=2592000
2014-08-11 09:13:44,696 INFO  crawl.AbstractFetchSchedule - maxInterval=7776000
2014-08-11 09:13:45,392 INFO  mapreduce.GoraRecordReader - gora.buffer.read.limit = 10000
2014-08-11 09:13:45,501 INFO  crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2014-08-11 09:13:45,501 INFO  crawl.AbstractFetchSchedule - defaultInterval=2592000
2014-08-11 09:13:45,501 INFO  crawl.AbstractFetchSchedule - maxInterval=7776000
2014-08-11 09:13:45,547 INFO  regex.RegexURLNormalizer - can't find rules for scope 'generate_host_count', using default
2014-08-11 09:13:45,654 INFO  mapreduce.GoraRecordWriter - gora.buffer.write.limit = 10000
2014-08-11 09:13:45,670 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2014-08-11 09:13:45,671 WARN  mapred.LocalJobRunner - job_local1662982347_0002
java.lang.NullPointerException
    at org.apache.avro.util.Utf8.<init>(Utf8.java:37)
    at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)

不幸的是，我不知道我在做什么錯。

我在“使用Apache Nutch進行Web爬網和數據挖掘”一書中實現了所有內容。

不幸的是返回錯誤。 目前，我很無知。

Answer 1

這是行失敗：

batchId = new Utf8(conf.get(GeneratorJob.BATCH_ID));

您的工作如何？ 如果我沒看錯，則不建議使用crawl命令，現在generate需要一個批處理id； 至少，這是我以前發生的事情。 現在有了開發分支，即使您未設置批處理ID似乎也可以正常工作...

---

從http://wiki.apache.org/nutch/Nutch2Tutorial ：

注意：不建議使用bin / nutch腳本中的檢索命令。 您應該使用單個命令，或者使用bin / crawl腳本...，它可以有效地將單個命令鏈接在一起。