automate downloading a file using c# programmatically

Question

I have a project where I need to automatically download and process an excel file from a public web site.

the site is the following:

http://apps.ahca.myflorida.com/dm_web/(S(rhlzd0ac2qwvvccbyp3lx2or))/doc_results_fo.aspx

you can see a link called Export Results which downloads the excel file. This link does a postback and I have been looking all over the place to find a way to automate it without success.

This is the last code I tried:

      try 
      {

                byte[] b = webClient.DownloadData("http://apps.ahca.myflorida.com/dm_web/(S(eha2oijpqo5mro1aywok4lly))/doc_results_fo.aspx");
                string s = System.Text.Encoding.UTF8.GetString(b);

                var __EVENTVALIDATION = ExtractVariable(s, "__EVENTVALIDATION");

                var forms = new NameValueCollection();

                forms["__EVENTTARGET"] = "lbtSpreadsheet";
                forms["__EVENTARGUMENT"] = "";
                forms["__VIEWSTATE"] = ExtractVariable(s, "__VIEWSTATE");
                forms["mTbdate"] = "11%2F15%2F2011";
                forms["__EVENTVALIDATION"] = __EVENTVALIDATION;

                webClient.Headers.Set(HttpRequestHeader.ContentType, "application/x-www-form-urlencoded");

                var responseData = webClient.UploadValues(@"http://apps.ahca.myflorida.com/dm_web/(S(eha2oijpqo5mro1aywok4lly))/doc_results_fo.aspx", "POST", forms);
                System.IO.File.WriteAllBytes(@"c:\tmp\FLORIDA.xls", responseData);


            }
            catch (Exception ex)
            {
               Console.Write(ex.StackTrace);
            }
        }


        private static string ExtractVariable(string s, string valueName)
        {
            string tokenStart = valueName + "\" value=\"";
            string tokenEnd = "\" />";

            int start = s.IndexOf(tokenStart) + tokenStart.Length;
            int length = s.IndexOf(tokenEnd, start) - start;
            return s.Substring(start, length);
        }

This is supposed to get the value of view state and other fields and issue a POST , but when I run it the file that gets downloaded is the page itself and not the excel file.

I am not sure if this is possible using WebClient, or I should use WebBrowser control (or similar controls), or maybe an atomated browsing tool that I can record a sequence of steps and run it every x days.

Any help will be greatly appreciated.

Thank you

Answer 1

I figured it out using Selenium .NET

        FirefoxProfile firefoxProfile = new FirefoxProfile();

                        firefoxProfile.SetPreference("browser.download.folderList", 2);
                        firefoxProfile.SetPreference("browser.download.dir", strFullpath);
                        firefoxProfile.SetPreference("browser.helperApps.neverAsk.openFile", "application/vnd.ms-Excel");
                        firefoxProfile.SetPreference("browser.helperApps.neverAsk.saveToDisk", "application/vnd.ms-Excel");
                        IWebDriver driver = new FirefoxDriver(firefoxProfile);

                        driver.Navigate().GoToUrl(link);
                        driver.FindElement(By.Id(lookup)).Click();
                        driver.Quit();