Java web scraper

Question

What is the best library for a Java web scraper? I know the following choices:

1. Selenium
2. HTMLUnit
3. Lobo browser

I need to select one option to build a scraper for one scalable project.

Answer 1

If you are scraping, why do you need a browser? Just doing basic cURL calls to a page and getting the response will give you what you need to do scraping.

This will help with scalability. If you want a browser then go for HTMLUnit as that would again help with scalability.

Answer 2

我最近推荐了Web Harvest ，并认为它开箱即用，除了围绕HTTP 500响应代码的一些问题...

Answer 3

Use jsoup , it works great to get the response from URL and then use the XPath Expression to parse data from the response. I've implemented this and it works great.