使用Eclipse创建Nutch 2.x插件

Question

I have to write a plugin to parse crawled content vy Nutch 2.3.1. I have decided to use eclipse as Its better than simple editor. Now How can I create a plugin in eclipse and test it via some simple use case ?

Answer 1

You can use following steps to get the plugin working from Eclipse.

1. Get Nutch source code.

   git clone https://github.com/apache/nutch.git

2. Switch to 2.3.1 branch. If you want latest 2.x in development, you can use 2.x branch https://github.com/apache/nutch/tree/branch-2.3.1

3. Import project in eclipse.

4. Build for eclipse. It uses ant for build and has eclipse target.

   ant eclipse

5. All available plugins in nutch are under src/plugins directory.

6. You need a smilar structure for your new plugin so copy one of the existing plugin to new one.

   cp -r lib-http my-http

7 Now check the structure of plugin directory. It should be as below.

my-http/
├── build.xml
├── ivy.xml
├── plugin.xml
└── src
    ├── java
    └── test

8. plugin.xml is the one which has definitions for extentions, extention-points, runtime libraries etc. You can view it in eclipse plugin editor and can do changes from there.

9. Add proper implementation class and tests. Map that for extension in plugin.xml

10. You have to change your build.xml and ivy.xml to add proper dependencies.

11. You can override targets defined in src/plugin/build-plugin.xml in your build.xml . build-plugin.xml is being called by main build file src/build.xml for each plugin.

12. You can test your plugin by using ant from the plugin directory. ant test .

You can also use eclipse to check JUint test results. Click on Test class and Run as JUnit Test

13. Add plugin to deploy and test targets in src/plugin/build.xml . This file is used by the main build file

    <ant dir="my-http" target="deploy"/>

14. add any required dependencies in build/ivy/ivy.xml

15. Add plugin plugin.includes property in conf/nutch-site.xml

16. Build nutch

    ant runtime

Now your plugin is set to run in local/distributed mode from runtime directory.

Answer 2

You can use any editor you want to write your code, as long as you generate a jar that you load in the Nutch plugin system with the right dependencies and configurations in the xml file everything should work. You can check https://wiki.apache.org/nutch/RunNutchInEclipse that contains detailed instructions to open and run within eclipse so debugging is easier, but its not required. Specially important is to run ant eclipse in your local copy of the project, so that you can open the Nutch entire source code in Eclipse, once this is done you can create your plugin file structure and start coding.

Hope it helps.