从使用Apache Nutch 1.4进行爬网和解析后获得的HTML文档中获取特定标签

Question

I used nutch 1.4 and crawled a website. I got the website crawled successfully and all the pages were dumped into segments. I merged all the segments to one segment and then i used readseg command to obtain a text version of all the crawled pages. Now I need to find out, URL of page and the meta data stored in that page. I don't know which command to use or shall i need to do something different.

Have made a lot of efforts on google Some people said that you have to write a separate plugin for it. Can someone tell me please.

Thanks a lot :) :)

Answer 1

Do a crawling. After that enter this into terminal.

bin/nutch readseg -dump crawl/segments/* output -nocontent -nofetch -nogenerate -noparse -noparsedata

If it runs, you will have a file with header informations plus contents in it. After that you can easily modify the file and get whatever info you want by string operations.

Answer 2

Finally, I am able to do it. Sharing in case someone else needs it. You can use index-metatags plugin provided here: http://wiki.apache.org/nutch/IndexMetatags

It will solve this problem Cheers :)