如何自定义 StormCrawler？

Question

Although I have been working with Apache Storm for a while, I am fairly new to StormCrawler.

I started a project from the StormCrawler+ES archetype. However, to customize StormCrawler, at what specific point should additional bolts be added?

Dave.

Answer 1

One way of doing would be to write a custom bolt and add it between the fetcher and the parser. It should look at the metadata for any mimetype given in the http response (bearing in mind the prefix used to store the info from the protocol ), possibly detecting the mimetype as done in the JSOUPParser . If it is an image then do your specific processing for it, then emit to the output. If it isn't, emit to a custom stream; the latter would be connected to the JSOUP parser so that you get outlinks; the former goes into ES.

You can find examples of dealing with the non-defaults streams in various places, in particular, the Tika module .