SSIS WebClient() Download File script Task issue

Question

I am wondering if someone could help me with this issue please. I am new to using script tasks in SSIS packages and i am unable to find solution to this issue. So i created this script Task to download a file from Web-portal. Package was working fine for couple of months but all of sudden, it started failing. Reason is that , url is being directed to Authentication page of Web-Portal again (looks like Client might have changed the security settings of their website or something). So now a blank file is being downloaded. below is the code i am using in my script task. Is there a code i can add to below script to by-pass the authentication page as i am already sending username and password in below script. One more thing to add, i can download the file manually when i copy and paste url in to Chrome so that means data file does exist in portal,it's just the script task being re-directed to authentication page again thus failing. Thanks in advance.

    public void Main()
    {
        // TODO: Add your code here

        WebClient wc = new WebClient();// { UseDefaultCredentials = true };
        var DownloadPath = Dts.Variables["User::varDownloadPathNew"].Value.ToString();
        DateTime startDate = DateTime.Parse(Dts.Variables["User::StartDate"].Value.ToString());
        DateTime enddate = DateTime.Parse(Dts.Variables["User::EndDate"].Value.ToString());

        wc.Credentials = new NetworkCredential("UserName", "Password");
        wc.DownloadFile("https://Test123.co.uk/model/download?&startfilter=" + startDate.ToString("dd") + "%2F" + startDate.ToString("MM") + "%2F" + startDate.ToString("yyyy") + "&mnu_jobdateendfilter=" + enddate.ToString("dd") + "%2F" + enddate.ToString("MM") + "%2F" + enddate.ToString("yyyy") + "&%A6&maxrows=400&format=excel&filename=Test.xls", DownloadPath);

        Dts.TaskResult = (int)ScriptResults.Success;

    }

Answer 1

Adding below code to my original code resolved the issue. As suggested by @billinkc it, portal was using cookies to Authenticate, so i checked cookie user name and password from ie/chrome (in my case, portal was using 2 cookie names and passwords so i used both) and then used it in below code wc.Headers.Add(HttpRequestHeader.Cookie,"cookiename=password");

public void Main()
{
    // TODO: Add your code here

    WebClient wc = new WebClient();// { UseDefaultCredentials = true };
    wc.Headers.Add(HttpRequestHeader.Cookie, "NSC_JOmbbd3tb4=ffffffffc3a03f7e45525; User=eyJhbGciOiJSU");
    var DownloadPath = Dts.Variables["User::varDownloadPathNew"].Value.ToString();
    DateTime startDate = DateTime.Parse(Dts.Variables["User::StartDate"].Value.ToString());
    DateTime enddate = DateTime.Parse(Dts.Variables["User::EndDate"].Value.ToString());

    wc.Credentials = new NetworkCredential("UserName", "Password");
    wc.DownloadFile("https://Test123.co.uk/model/download?&startfilter=" + startDate.ToString("dd") + "%2F" + startDate.ToString("MM") + "%2F" + startDate.ToString("yyyy") + "&mnu_jobdateendfilter=" + enddate.ToString("dd") + "%2F" + enddate.ToString("MM") + "%2F" + enddate.ToString("yyyy") + "&%A6&maxrows=400&format=excel&filename=Test.xls", DownloadPath);

    Dts.TaskResult = (int)ScriptResults.Success;

}