[英]Apache Nutch: FetcherJob throws NoSuchElementException deep in Gora
我正在运行Apache Nutch 2.3.1开箱即用,它使用的是Gora 0.6.1。 我按照这里的说明操作: http : //wiki.apache.org/nutch/RunNutchInEclipse
它与InjectorJob
运行良好。
现在我正在运行FetcherJob
,Gora使用MemStore
作为数据存储。 我有gora.properties
包含
gora.datastore.default=org.apache.gora.memory.store.MemStore
抛出:
2016-10-02 22:55:54,605 ERROR mapreduce.GoraRecordReader (GoraRecordReader.java:nextKeyValue(121)) - Error reading Gora records: null
2016-10-02 22:55:54,605 INFO mapred.MapTask (MapTask.java:flush(1460)) - Starting flush of map output
2016-10-02 22:55:54,614 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2016-10-02 22:55:54,615 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local874667143_0001
java.lang.Exception: java.lang.RuntimeException: java.util.NoSuchElementException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.RuntimeException: java.util.NoSuchElementException
at org.apache.gora.mapreduce.GoraRecordReader.nextKeyValue(GoraRecordReader.java:122)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
at java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
at org.apache.gora.memory.store.MemStore.execute(MemStore.java:128)
at org.apache.gora.query.impl.QueryBase.execute(QueryBase.java:73)
at org.apache.gora.mapreduce.GoraRecordReader.executeQuery(GoraRecordReader.java:67)
at org.apache.gora.mapreduce.GoraRecordReader.nextKeyValue(GoraRecordReader.java:109)
... 12 more
2016-10-02 22:55:55,383 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_local874667143_0001 running in uber mode : false
2016-10-02 22:55:55,385 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
2016-10-02 22:55:55,387 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Job job_local874667143_0001 failed with state FAILED due to: NA
2016-10-02 22:55:55,396 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0
Exception in thread "main" java.lang.RuntimeException: job failed: name=, jobid=job_local874667143_0001
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:119)
at org.apache.nutch.fetcher.FetcherJob.run(FetcherJob.java:205)
at org.apache.nutch.fetcher.FetcherJob.fetch(FetcherJob.java:251)
at org.apache.nutch.fetcher.FetcherJob.run(FetcherJob.java:314)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.fetcher.FetcherJob.main(FetcherJob.java:321)
这种情况发生在Nutch和Gora的深处,我不知道它为什么会发生。 我尝试升级到Gora 0.8但同样的问题。 我尝试将Gora降级到0.6,同样的问题。 我想切换到另一个像hBase这样的数据存储,但这对我目前需要的东西来说有点过分。
请帮我解决这个问题。
我确认问题出在MemStore中。
在0.6.1中有一个错误: https : //github.com/apache/gora/blob/apache-gora-0.6.1/gora-core/src/main/java/org/apache/gora/memory/store /MemStore.java#L128
这已经在master中解决了: https : //github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/memory/store/MemStore.java#L155,the访问#firstKey()有一个警卫#isEmpty()
但是,不要尝试更新到Gora 0.7-SNAPSHOT,因为Nutch现在还没有适应它。
如果你想使用Gora-0.7-SNAPSHOT和Nutch 2.x,也许你可以让它工作:
mvn install
/将其安装在maven的本地存储库中 我希望它有效:)
关于使用HBase,可以很容易地进行本地安装进行实验。
/home/you/hbase
到目录中,例如: /home/you/hbase
cd /home/you/hbase/bin
./start-hbase.sh
现在你已经启动并运行了HBase。 配置Nutch:
ivy / ivy.xml:看看@ Emmanuel关于HBase的常春藤依赖配置的评论。
gora.properties:
gora.datastore.default=org.apache.gora.hbase.store.HBaseStore
gora.datastore.autocreateschema=true
gora.datastore.scanner.caching=100
Nutch的-site.xml中:
<configuration>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
<description>Default class for storing data</description>
</property>
</configuration>
完成。 它将采用HBase的所有默认配置:localhost,/ tmp / ...,blablabla
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.