如何仅使用 c# Web 浏览器加载 html

Question

I'm using C# web browser to scrape data from a website. The problem is that it takes around 20 minutes to get around 250 records.

What I do programmatically is

1- Get all years inside the dropdown.

2- For each year I make a search. and scrap data from the table.

3- First cell of the row is a link (details) and rest of the cells have basic information. So what I do is get the basic information, open the details link in a new browser and get the details.

4- loop through step 3.

I made a performance test to the program and saw that it takes a lot of time for waiting the document to load. If I skip scraping data from the details page it takes 1.5 minutes to scrap all the data. I use the following method to wait for the document to complete before I start scrapping.

public async Task WaitPageLoad(int timeOut)
{
    var pageLoaded = new TaskCompletionSource<bool>();
    var timeElapsed = 0;
    DocumentCompleted += (s, e) =>
    {
        if (ReadyState != WebBrowserReadyState.Complete) return;
        if (pageLoaded.Task.IsCompleted) return; pageLoaded.SetResult(true);
    };

    while (pageLoaded.Task.Status != TaskStatus.RanToCompletion)
    {
        await Task.Delay(10); 
        timeElapsed++;
        if (timeElapsed >= timeOut * 100) pageLoaded.TrySetResult(true);
    }
}

So I was wondering if there's any way to make the browser to only load html and not images or something.

Any help is very appreciated!

Answer 1

Why use WebBrowser at all? This is a control used to parse and display content to users. That's not quick by any stretch.

If all you want is the data (and don't intend to display it) you could simply do something like:

//Gets you the HTML for a given URL synchronously
var data = new System.Net.WebClient().DownloadString(url);

However the above can be more difficult to use depending on the complexity of the page(s) you're trying to scrape.

For more advanced web scraping I'd recommend grabbing either HtmlAgilityPack or IronWebScraper from NuGet.

Answer 2

Depending on how you login to the website, you need something like this to work with WebClient:

WebClient client = new WebClient();
client.Credentials = new NetworkCredential("Username", "Password");
string pageData = client.DownloadString("https://stackoverflow.com/");