如何在Nutch 2.3中抓取图像作为HBase作为后端？

Question

I want to crawl images from certain sites. So far I tried modifiying regex-urlfilter.txt.

I changed:

 -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PP
 T|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$

To:

-\.(css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|t
gz|TGZ|mov|MOV|exe|EXE|js|JS)$

But it didn't work. I am surprised that I didn't find any documentation regarding crawling images using Nutch 2.3 . Referal to any existing documentation would really be a great help.

Answer 1

In order to fetch and store images using Nutch you have to follow these steps:

1- Adding regular expression to not filter image formats, such as jpg, jpeg, tif, gif, png and etc... (which you already did)

2- Implementing a parse plugin for parsing images. For more information about Nutch extension points and writing required plugin follow these links:

http://wiki.apache.org/nutch/AboutPlugins

http://wiki.apache.org/nutch/WritingPluginExample

3- Tell Nutch about the implemented plugin and using that for image file formats:

For this purpose you have to follow two different steps, first, modify conf/parse-plugins.xml and map your implemented plugin to image file formats:

<mimeType name="image/jpeg">
        <plugin id="parse-image" />
</mimeType>
<mimeType name="image/gif">
        <plugin id="parse-image" />
</mimeType>
<mimeType name="image/png">
        <plugin id="parse-image" />
</mimeType>

second, add the implemented plugin to nutch-site.xml to be run at Nutch runtime. You have to add the implemented plugin to <plugin.includes> property.