如何使用Webclient / HttpClient以编程方式获取数据？

Question

I wanted to download the data from https://eauction.ccmc.gov.in/frm_scduled_items.aspx using the date listed in the dropdown.

    private async Task Cbetest()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("https://eauction.ccmc.gov.in");
            var content = new FormUrlEncodedContent(new[]
        {
            new KeyValuePair<string, string>("ctl00$ContentPlaceHolder1$gridedit$ctl14$ctl02","17/02/2016")
        });
            var result = await client.PostAsync("/frm_scduled_items.aspx", content);
            string resultContent = await result.Content.ReadAsStringAsync();
            Console.WriteLine(resultContent);
        }
    }

I wanted to download the data shown in the above image

Answer 1

You need to do a little extra work to simulate a post to begin scraping against a ASP.NET WebForms application. Mostly, you're going to need to pass along valid ViewState and EventValidation parameters, which you can retrieve from an initial GET request.

I'm using the HTML Agility Pack to ease with parsing the initial response, I recommend you look into it if you're planning to scrape HTML.

The following seems to get the results you're looking for, though I haven't looked too deeply in the response HTML.

using(var client = new HttpClient())
{
    client.BaseAddress = new Uri("https://eauction.ccmc.gov.in");

    var initial = await client.GetAsync("/frm_scduled_items.aspx");

    var initialContent = await initial.Content.ReadAsStringAsync();

    var htmlDoc = new HtmlDocument();

    htmlDoc.LoadHtml(initialContent);

    var viewState = htmlDoc.DocumentNode.SelectSingleNode("//input[@id='__VIEWSTATE']").GetAttributeValue("value", string.Empty);
    var eventValidation = htmlDoc.DocumentNode.SelectSingleNode("//input[@id='__EVENTVALIDATION']").GetAttributeValue("value", string.Empty);

    var content = new FormUrlEncodedContent(new Dictionary<string, string>{
        {"__VIEWSTATE", viewState},
        {"__EVENTVALIDATION", eventValidation},
        {"ctl00$ContentPlaceHolder1$drp_auction_date", "17/02/2016"}
    });

    var res = await client.PostAsync("/frm_scduled_items.aspx", content);

    var resContent = await res.Content.ReadAsStringAsync();

    Console.WriteLine(resContent);
}

From there you'll want to parse the resulting table to get useful information. If you want to crawl through the DataGrid's pages, you're going to need to get updated EventValidation and ViewState values and simulate additional posts for each page.