WCF“自托管”应用程序变得无法响应

Question

We have a C# (.Net 4.0) console application that "self hosts" two WCFs services: one used WSHttpBinding , and another uses BasicHttpBinding .

Connecting to these services, we have two separate client applications: a Silverlight-based service that uses the BasicHttpBinding , and another console app that uses the WSHttpBinding .

The WCF service app usually has around 30 users connected via the Silverlight client, and another couple of connections from the console application client. It's not "flat out" by any means; each client queries the WCF service maybe once every 5 seconds at the very most.

The problem is: intermittently the service application becomes unresponsive. Although the server itself continues to run (it continues to write to a log file), all WCF activity (on both ServiceHost s) appear to "seize". New requests aren't processed (although the TCP connections are accepted). Also, the number of threads consumed by the application starts to ramp up dramatically, at the rate of around one new thread per second. The code itself doesn't do anything with Thread s or ThreadPool s, although it occasionally will issue a Thread.Sleep for a few hundred milliseconds.

The frustrating thing is the intermittent nature of the problem: the code regularly runs for hours, even days without any issues. Then, with no apparent cause, it suddenly becomes unresponsive and the Thread count starts ticking up.

I've tried simulating user activity - connecting and disconnecting clients, "swamping" the service with requests - but I can do nothing to reproduce the fault.

Just in case the issue was WCF Throttling, I've added this code:

 ServiceThrottlingBehavior throttlingBehavior = new System.ServiceModel.Description.ServiceThrottlingBehavior
                                                           {
                                                               MaxConcurrentCalls = 512,
                                                               MaxConcurrentInstances = 8192,
                                                               MaxConcurrentSessions = 8192
                                                           };

        host.Description.Behaviors.Add(throttlingBehavior);
        host2.Description.Behaviors.Add(throttlingBehavior);

.. with no apparent effect.

I've put extensive logging in the code to try and determine what it is that triggers this behaviour - logging each call to each method - but nothing has appeared as a result. I've wrapped everything in try ... catch blocks and spitting any exceptions to the log file, to see if something's falling over somewhere, and also trapped the UnhandledException s in a similar fashion ... but again, nothing appears to be going wrong.

Does the above behaviour sound familiar to anyone, or can anyone suggest on the best way forward to troubleshoot this issue?

EDIT: following Wal's advice below, I've captured a .DMP of the application when it starts mis-behaving, and looking at the Parallel Stacks view in VS2012, I see:

... and others very similar but with different numbers of threads. I'm not clever enough to decode exactly what this means.. can anyone suggest where to start looking next?

Answer 1

what is the concurrencymode for the service? and the instancecontextmode?

the default instancecontextmode is per session, it may be worth changing this to percall, this will use more memory but will ensure that each service instance is no hanging around (provided the client is correctly disposed of http://coding.abel.nu/2012/02/using-and-disposing-of-wcf-clients/ )

Answer 2

As pointed out sooner, sounds like you have a race condition. Aren't you by any chance checking System.ServiceModel.ICommunicationObject.State of the connection somewhere in the code? See MSDN article :

Checking the value of the System.ServiceModel.ICommunicationObject.State property is 
a race condition and is not recommended to determine whether to reuse or close a channel.

Answer 3

Thank you to everyone who's commented and answered; your suggestions and input have really helped - not least to confirm that it doesn't seem to be something trivial I've missed.

However, and slightly frustratingly, the problem appears to have gone away. Here's what I've changed:

• The application was writing to the console regularly (my "WriteToLog" method had Console.WriteLine as well as appending to a file; this was purely for my own convenience during development). The application was also running as a service using FireDaemon, and for some reason we started to see high CPU time on conhost.exe . So to counteract that, I've commented out the Console.WriteLine .

• Due to the high CPU, we also increased the performance of the Virtual Machine the code was running on by throwing a few more cores at it.

As a result, the application is now much "quieter" in terms of CPU usage. As others have mentioned, there's almost certainly a "race condition" in the code somewhere, but by making the underlying machine faster and the code more efficient, it almost seems as if I've decreased the chances of the race condition occurring. Certainly, the problem that was occurring at least once a day hasn't happened in nearly a week.

Just to be sure, I've gone through the code and made sure each shared object is wrapped in a Lock() where there's any possibility of it being modified by another thread - even though I'm not doing any explicit threading, I'm assuming that the WCF mechanism will automatically do that and there's the potential for an incoming request to try and modify an object whilst something else is chewing on it. I would have expected some kind of concurrency exception if that happened though?

Thanks again for the help, and here's hoping the code doesn't fall over just after I click the Post Your Answer button :/

Answer 4

May be its a threading issue which has nothing to do with WCF - as mentioned in previous posts, lock statements may be the suspect - your application (WCF part or not), may have spun up threads, which cannot exit due to locking issue.

On the other hand, it could be WCF, did you receive a lot of hit on your WCF service? Try throttle it. http://msdn.microsoft.com/en-us/library/system.servicemodel.description.servicethrottlingbehavior.maxconcurrentinstances.aspx