WebClient Timing out when Scheduled in Task

Question

Hey guys this is my first attempt at using the Task libraries in 4.0 so if you see anything else I'm besides my problem that is not correct please do let me know.

My problem is that when I schedule a bunch of task which inside use a webclient to make a request, the first few make it through just fine, but after a certain time my webclient starts throwing an exception. It as if it creates the webclient then sticks in the Task and waits for a thread to pick it up but by that time the timeout time is reached .. that's just my assumption.

Here is the code :

var TmsThread = Task.Factory.StartNew(() => UpdateTmsNullPackages(), TaskCreationOptions.LongRunning);

that runs in the Form1_Load of the windows app. This is what it calls

public void UpdateTmsNullPackages()
    {
        Parallel.ForEach(TmsNullPackages, Package =>
        {
            try
            {
                Task<string> task = Task.Factory.StartNew(() => Package.GetPackageTmsId(), TaskCreationOptions.AttachedToParent);
                task.ContinueWith(t =>
                    {
                        if (!String.IsNullOrEmpty(t.Result))
                        {
                            Package.TMSID = t.Result;
                            NowTmsIdFoundPackages.Add(Package);
                        }
                    });
            }
            catch(Exception ex){}
        });
    }

which in turn, runs this

 public static string GetPackageTmsId(this TwcPackage Package)
    {
        string TMSID = null;
        if (!(String.IsNullOrEmpty(Package.movie_Provider)) && !(String.IsNullOrEmpty(Package.movie_Product)) && !(String.IsNullOrEmpty(Package.movie_Provider_ID)) && !(String.IsNullOrEmpty(Package.movie_Asset_ID)))
        {
            try
            {
                using (WebClient client = new WebClient())
                {
                    client.Credentials = new NetworkCredential(TMSID_Recheck.Properties.Settings.Default.WebRequestUser, TMSID_Recheck.Properties.Settings.Default.WebRequestProdUserPassWord);
                    XmlDocument xmlDoc = new XmlDocument();
                    string URLToBeRequested = TMSID_Recheck.Properties.Settings.Default.RequestProdBaseURL + TMSID_Recheck.Properties.Settings.Default.RequestAPIVersion + "/" + TMSID_Recheck.Properties.Settings.Default.RequestAPIProgramServiceCall + TMSID_Recheck.Properties.Settings.Default.RequestAPIProgramAssociationServiceCall + Package.movie_Provider + ':' + Package.movie_Product + ':' + Package.movie_Provider_ID + "::" + Package.movie_Asset_ID;
                    try
                    {
                        xmlDoc.LoadXml(client.DownloadString(URLToBeRequested));
                        XmlNodeList Program = xmlDoc.DocumentElement.SelectNodes("program");
                        if (Program.Count > 0) TMSID = Program[0].Attributes["TMSId"].Value.ToString();
                    }
                    catch (WebException ex)
                    {
                        if (ex.Status != WebExceptionStatus.Timeout)
                        {
                            if (((HttpWebResponse)ex.Response).StatusCode != HttpStatusCode.NotFound) {  }
                        }
                        else {  }
                    }
                }
            }
            catch (Exception ix) {  }
        }
        return TMSID;
    }

the issue happens when downloadstring is called after a couple hundred tasks it throws a timeout exception.

Answer 1

the issue happens when downloadstring is called after a couple hundred tasks

And how many tasks have completed in that time slice?

It looks like you are simply queuing too many requests. Your system and the remote server probably have policies in place to limit the number of simultaneous connections.

The solution (and a quick diagnosis test) would be to use MaxDegreeOfParallelism in the ForEach.

Here is a similar question with some good answers.