NavigateToPage hangs in ASP.NET application

Question

I'm writing a web scraper using ScrapySharp that needs to use a simulated browser to log in and access data. The method, with the NavigateToPage call, works in a console application but not in my ASP.Net application. No NavigateToPage calls go through, including attempted navigations to Google, with or without https, with or without the default cookies parser.

 public static ScrapingBrowser GetESEBrowser()
        {
            ScrapingBrowser browser = new ScrapingBrowser();
            browser.UseDefaultCookiesParser = false;
            WebPage ESE = browser.NavigateToPage(new Uri("http://www.ese-co.com/storefrontCommerce/login.do"));

            PageWebForm login_form = ESE.FindForm("loginForm");
            login_form["usr_name"] = "blahblah";
            login_form["usr_password"] = "blahblah";

            login_form.Submit();

            return browser;
        }

Answer 1

There seems to be a deadlock in HttpRequest.GetResponseAsync, called in ScrapingBrowser.cs:272. It seems to be related to the problem as described here: https://blog.stephencleary.com/2012/07/dont-block-on-async-code.html

Try an async Task like this to replace your GetESEBrowser():

public static async Task<ScrapingBrowser> GetESEBrowser()
        {
            ScrapingBrowser browser = new ScrapingBrowser();
            WebPage ESE = await browser.NavigateToPageAsync(new Uri("https://www.test.com"));

            //...

            return browser;
        }

You can call the Task with

GetESEBrowser().ConfigureAwait(false);