Java Web 爬虫和爬虫

Question

My intention is to read cost details of a product from various websites , so that i can display cost comparison details in a html page of my Spring application. Can anyone suggest me on how to do it . is there any technologies to achieve this ? so that i can always read the updated data from other websites and display it in my Spring application. I saw some Web scraper tools as a Chrome extension but it generates an Excel workbook. how could i use it in my Spring application and display it in HTML page ?

Answer 1

You can send http queries from your spring application and parse answers updating data. Or you can use any external tool that will scrape whatever you want and to save results( for example as an Excel workbook), and your application will read this results and process it however you want.

Answer 2

There are a lot of opensource Java and python based crawler readily available which you can configure for your requirement, some of which are as stated below.

Apache Nutch
StormCrawler
Jsoup
Jaunt

in your case, since you need the only price from the product page you can build your own using JSoup a framework available in Java or Beautiful Soup module in Python.

if the scale isn't a concern and you just want to crawl some pages on a daily basis I recommend building your own crawler. otherwise, you can use Nutch or StormCrawler

Also for custom made please don't have multiple selectors for different webpages, in fact just find out a common tag, CSS or template which will get you the price.