Web scrape without use of browser control?

Question

I have a SharePoint site that I currently have a webscrape tool using written in VB and using the web browser control. It's unbelievably slow and I am looking to update it. Can we use some toolkits such as HTMLAgilityPack to load a document into DOM, and paste/click buttons just using the html elements rather than having to load the actual web page to handle it?

Answer 1

One possible solution would be to use a Nuget package called selenium web driver. What this package does is controls your browser (IE, Firefox, Safari, Chrome, etc) and it will act just like a web browser control. Using selenium may be just as slow as a web browser control in some instances. You mentioned using a package such as HTML agility pack; this would work but where you are wrong is in the fact that it would still be loading the page; you just dont see it.

Here is a link on how to use selenium