Limitations of multiple XDocument.Load(uri) requests?

Question

I'm threadpooling just under 400 XDocument.Load() requests to similar but unique URIs over the Internet. Everytime time I do it, 32 of the requests cause an "Root element is missing." exception. It's different requests each time that cause the exception but there is always 32, which is slightly suspicious. The XML always varies in length but none are huge < ~ 16kb.

Is there some sort of limit in XDocument.Load() that is causing this error? Or perhaps the underlying WebRequest (if that is what is being used) or the internal XmlReader? Or perhaps the threading? I'm adding QueueUserWorkItem to add the requests to the thread pool.

Any ideas?

Many thanks.

Answer 1

Personally I doubt that this is XDocument. The first thing I would do is attach a network tool such as Fiddler to see what came over the wire. In particular I wonder if you are being throttled by a NLB or similar.

If nothing obvious in there, I would split it into step steps; download as XML (maybe WebClient DownloadString) and then parse the XML locally. The advantage here is that you can add some try/catch so that if/when it fails you log the response string you got, for analysis.

If you do this, and it starts behaving, then that might suggest a bug in XDocument.Load; likewise if you do this and it fails but all responses were well-formed XML, then similar - a bug in Parse. But it should also rule out (or identify) a network-related cause.

Answer 2

• Wrap each Load() in a re-try loop, say, 3 times.
• Reduce max threads in ThreadPool to something sensible like 16. http://msdn.microsoft.com/en-us/library/system.threading.threadpool.setmaxthreads.aspx
• Use ab to determine if the remote server is unreliable. http://httpd.apache.org/docs/2.0/programs/ab.html