How can I make an application in c# collect data from a website?

Question

First of all, I hope my question doesn't bother you. I really need to get and idea of how I can accomplish that, but unfortunatelly, I'm really a beginner, I'm crawling when it comes to programming. I'm struggling to learn it the best way I can. I'll thank you for any help you give me.

Here's the task: I was ordered to find a way to collect some data from a website using ac# application. This will be done everyday, in order to update the data which we'll use to calculate some financial index.

I know my question might sound vague, anyway, even telling me how I can be more precise will help me. I know I seem to know desperate, but putting appart all the personell issues, my scholarship kind of depends on it.

Thanks in advance! (Please, don't mind the bad English, I'm brasilian and my English might not be that good yet.)

Answer 1

First, your English is fine. In fact, I thought you were a native speaker until you said otherwise.

The term you're looking for is 'site scraping'. Observe this question: Options for HTML scraping? . The second answer points to an HTML agility pack library you can use.

Now, there are two possibilities here. The first is you have to parse the HTML and scrape your data out of it. This is more computationally intensive and depends on the layout of the page. If they change the way the site looks, it could break the scraper.

The second possibility is they provide some XML or JSON web service you can consume. In this case you aren't scraping anything, but are rather using a true data feed. If the layout of the site changes, you will not break. Whether your target site supports this form of data feed is up to the site.

Answer 2

If I understand your question, you're being asked to do some Web Scraping , where you 1) download the contents of a web page and 2) try to parse data from that content.

For step #1, you should look into using a WebClient object in C# to download the HTML from the web page. You can give a WebClient object the URL you want to download the content from and obtain a String containing the content (probably HTML) of the URL.

How you go about doing step #2 depends on what content is present at the web site. If you know of certain patterns you're looking for in the HTML, you can search the HTML string using various methods. A more general solution for parsing HTML data can be found through using the Html Agility Pack , which will let you handle the HTML as a tree structure (DOM).

Answer 3

Use the WebClient class to get the page.

Turn the html into xml.

Use XPath to select the data you are interested in.

Answer 4

Ok, this is a pretty straightforward app design, and a lot of the code exists that you can reuse. Since you're a beginner, I'll break down into steps of what you need to do and recommend approaches.

1) You will use classes from System.Net to pull the web pages (WebClient being the easiest to usse). You will want to have this part of the program run on a timer if you can (using the scheduled jobs feature of the OS) and have it just pull the pages and drop them in a folder.

2) You have a second job which will run separately, pulling unread files from that folder, parsing them (using the HtmlAgility pack library is best) and then storing them in an index of some kind ( Lucene is best for that)

3) You have a front end application of some sort (web or desktop) which queries that index for the information you're looking for.