HttpWebRequest returns different results than web browser

Question

I am trying to crawl a web site and fetch its pages' data. It all went well till one point where I started to get different data than the one coming in the view-source (the one I have expected). I am using html-agility-pack but also tried using HttpWebRequest, in order to change all it's headers, but no luck. I know it isn't a JavaScript trick because of the data in the view source. I hope someone has an idea to a possible explanation or solution.

Answer 1

There are a lot of factors that could possibly affect the html sent to you by the server. I suggest you use a tool such as fiddler to inspect the request headers more closely.

From my experience the usual culprits in such problems are cookies and session information stored as hidden fields in the document and sent with each request as part of the request body.