C# - Downloading website with webcrawler for ajax page

Question

I've written a simple Web crawler project in c# (using Microsoft WebCrawler) that download the content of a specific site.

I've registered to the event for DocumentCompleted and waiting for the ReadyState of the page to be Complete.

When i'm navigating to the page using any browser the site works perfectly fine. When I'm trying to do the the same throughout my code the ajax part of the page is not loading correctly.

As Marcom suggested I looked at the response with fiddler, both from my web crawler and from a web browser. When my web crawler navigates to the web page he receives false on one of the params in contrary of the web page.

Any lead?

Answer 1

you will probably have to simulate the requests done by ajax to download the rest of the data.

Use a tool like fiddler to see what is going on behind the scenes.