如何自定义Apache Nutch 2.3生成步骤

Question

I want Nutch to select specific URLs according to my own rules. This step is done at generate time. I know how to write parser/indexer plugin. But How to do it at generate time. My Nutch version is 2.3 series

Answer 1

The Nutch generator is not really an extension point in Nutch, so you are not able of writing plugins to customize it. Nevertheless, nothing stops you from writing your own generator with your own logic.

You would need to adjust the bin/nutch and bin/crawl scripts in order to call your own generator instead of the default one. Keep in mind that some other parts of Nutch rely on some parts of the generator implementation ( SegmentMerger for instance). If you customize these parts, then you'll need to update some other classes as well.

The generator uses the ScoringFilter.generatorSortValue() method when is deciding which elements to return. So, this is one alternative that doesn't require changing the generator.

Side note, this is not entirely uncommon to do, I've seemed some clients requiring customized generators.

Answer 2

As suggested by Jorge, you could write a scoringfilter to assign scores to pages based on your own logic and filter during the generation step based on that. Alternatively, if by chance your selection rules can be determined based on the URL alone, you could have a bespoke URL normaliser used with a scope of generate (or whatever the value is) which would rewrite the URLs into something that the URL filters would then discard. You'd need to activate the filtering as part of the generate step. This is an ugly hack.

Nutch 2.x is really awkward and I am not sure you could create a copy of your table based on a filter of the original one.

What Gora backend do you use?

StormCrawler is a lot more flexible for this and we've recently added a mechanism for filtering URLs at the spout level , which is exactly what you'd need. You could do a similar thing in Nutch 2.x but that would probably mean changing things in GORA as well.