C# Using HTML Agility Pack and Need To programmatically Login To Website

Question

Using HTML Agility Pack, I can get the HTML document of the non-member version of the site and parse what I want from it.

var web = new HtmlWeb();
var financialPage = web.Load("http://www.gurufocus.com/financials/ko");

I'm a member of the site. I have a username and password and want to programmatically login in and scrape all the data that is only available to members.

I've spent a lot of time reading answers like this... How to programmatically log in to a website to screenscape? and I just can't figure it out. Can't someone please explain to me what I need to do get the member's version of the HTML document? Thank you.

Answer 1

It took me a long time to come to grips with the fact that this doesn't work with HTML Agility Pack. Just use Selenium with PhantomJS. It solved all my problems and it's possible to use CSS selectors. Don't waste as much time as I did on this.