Web scraping: How to get specific element from some page?

Question

I'm developing some scraping software to check if products price are changed or not. To do this I'm sending Web http request to get product page, it returns me whole html page. There are a lot of items to check so I'm doing it in parallel so my internet bandwidth runs out quickly

The main questions:

Is it possible to send Web http request with select query to get only specific element from html page ?

Answer 1

Your best (free / cheap) bet is the Html Agility Pack. There are automated solutions out there that have a UI where you stick in the URL and then just point and click to the elements that you want to extract and they generate the info for you. Big $$$ for that stuff though :(. With the Html Agility Pack, you have to create the "xpath" by hand.

Answer 2

You can't unless the target server supports Range Request. You can send a request to test if the target server accents range request.