[英]Installing Apache Nutch on Windows
I'm trying to integrate Apache Solr with Apache Nutch 1.14 on Windows 7 (64 bit), but I'm getting an error while trying to run Nutch. 我正在尝试将Apache Solr与Windows 7(64位)上的Apache Nutch 1.14集成,但是在尝试运行Nutch时出现错误。
Things I already did: 我已经做过的事情:
(I tried the Hadoop WinUtils 2.7.1 with no success too). (我尝试了Hadoop WinUtils 2.7.1,也没有成功)。
The error I'm getting: 我得到的错误:
$ bin/crawl -i -D http://localhost:8983/solr/ -s urls/ TestCrawl 2
Injecting seed URLs
/home/apache-nutch-1.14/bin/nutch inject TestCrawl/crawldb urls/
Injector: starting at 2018-06-20 07:14:47
Injector: crawlDb: TestCrawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:609)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:125)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at org.apache.nutch.crawl.Injector.inject(Injector.java:417)
at org.apache.nutch.crawl.Injector.run(Injector.java:563)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.Injector.main(Injector.java:528)
Error running:
/home/apache-nutch-1.14/bin/nutch inject TestCrawl/crawldb urls/
Failed with exit value 1.
After downloading the hadoop-core-1.1.2.jar file from: http://www.java2s.com/Code/Jar/h/Downloadhadoopcore121jar.htm , and pasting it in the NUTCH_HOME/lib folder, I'm getting the following error: 从http://www.java2s.com/Code/Jar/h/Downloadhadoopcore121jar.htm下载hadoop-core-1.1.2.jar文件并将其粘贴到NUTCH_HOME / lib文件夹后,我得到了以下错误:
$ bin/crawl -i -D http://localhost:8983/solr/ -s urls/ TestCrawl 2
Injecting seed URLs
/home/apache-nutch-1.14/bin/nutch inject TestCrawl/crawldb urls/
Injector: starting at 2018-06-20 23:19:49
Injector: crawlDb: TestCrawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.getInstance(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)Lorg/apache/hadoop/mapreduce/Job;
at org.apache.nutch.crawl.Injector.inject(Injector.java:401)
at org.apache.nutch.crawl.Injector.run(Injector.java:563)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.Injector.main(Injector.java:528)
Error running:
/home/apache-nutch-1.14/bin/nutch inject TestCrawl/crawldb urls/
Failed with exit value 1.
If I don't set the HADOOP_HOME variable, I'm getting the following exception: 如果我没有设置HADOOP_HOME变量,则会收到以下异常:
Injector: java.io.IOException: (null) entry in command string: null chmod 0644 C:\cygwin64\home\apache-nutch-1.14\TestCrawl\crawldb\.locked
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:869)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:852)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:854)
at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1154)
at org.apache.nutch.util.LockUtil.createLockFile(LockUtil.java:59)
at org.apache.nutch.util.LockUtil.createLockFile(LockUtil.java:81)
at org.apache.nutch.crawl.CrawlDb.lock(CrawlDb.java:178)
at org.apache.nutch.crawl.Injector.inject(Injector.java:398)
at org.apache.nutch.crawl.Injector.run(Injector.java:563)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.Injector.main(Injector.java:528)
Error running:
/home/apache-nutch-1.14/bin/nutch inject TestCrawl//crawldb urls/
Failed with exit value 127.
I would really appreciate any help I can get! 我会很感激我能得到的任何帮助!
When you execute Crawl just execute this following command 当您执行抓取时,只需执行以下命令
bin/crawl -s urls/ TestCrawl/ 2
And after you can use this (-D with class) 然后您可以使用它(-D与类)
bin/nutch index -Dsolr.server.url=http://localhost:8983/solr/YOURCORE TestCrawl/crawldb/ -linkdb TestCrawl/linkdb/ TestCrawl/segments/* -filter -normalize -deleteGone
Or you can specify in conf/nutch-site.xml 或者您可以在conf / nutch-site.xml中指定
<property>
<name>solr.server.url</name>
<value>http://localhost:8983/solr/YOURCORE/</value>
<description>Defines the Solr URL into which data should be indexed using the indexer-solr plugin.</description>
</property>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.