Debugging NullReference exception from WCF client internals

Question

We have a WCF service and roughly the following client code:

bool success = false;
IClientChannel proxy = null;
try
{
  var client = channelFactory.CreateChannel(new EndpointAddress(url));
  proxy = client as IClientChannel;
  proxy.Open();
  client.MyWcfMethod(a, b, c);                                
  proxy.Close();
  success = true;
}
finally
{
  if (!success)
    proxy.Abort();
}

It seems to work fine, but on our beta environment, once in ~70000 calls a NullReferenceException is thrown. We log the exception and all the data around carefully, and it looks like it does not depend on data in any way. If I repeat the call immediately after the exception (just a while until success), it goes smoothly.

The exception appears so rare, that I can't reproduce, trace or debug it. Exception seems to be thrown immediately at the time of call, no server code is called.

Here is the stack trace:

Object reference not set to an instance of an object. Server stack trace: 
at System.Xml.Linq.ElementWriter.WriteElement(XElement e) 
at System.Xml.Linq.XElement.WriteTo(XmlWriter writer) 
at System.Xml.Linq.XElement.System.Xml.Serialization.IXmlSerializable.WriteXml(XmlWriter writer) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteIXmlSerializable(XmlWriterDelegator xmlWriter, Object obj, XmlSerializableWriter xmlSerializableWriter) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteIXmlSerializable(XmlWriterDelegator xmlWriter, Object obj) 
at System.Runtime.Serialization.XmlDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator xmlWriter, Object obj, Boolean isDeclaredType, Boolean writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle) 
at WriteCallContextToXml(XmlWriterDelegator , Object , XmlObjectSerializerWriteContext , ClassDataContract ) 
at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle) 
at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle) 
at System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver) 
at System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph) 
at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectContentHandleExceptions(XmlWriterDelegator writer, Object graph) 
at System.Runtime.Serialization.DataContractSerializer.WriteObjectContent(XmlDictionaryWriter writer, Object graph) 
at System.ServiceModel.Channels.XmlObjectSerializerHeader.OnWriteHeaderContents(XmlDictionaryWriter writer, MessageVersion messageVersion) 
at System.ServiceModel.Channels.MessageHeader.WriteHeaderContents(XmlDictionaryWriter writer, MessageVersion messageVersion) 
at System.ServiceModel.Channels.MessageHeaders.WriteHeaderContents(Int32 headerIndex, XmlDictionaryWriter writer) 
at System.ServiceModel.Channels.Message.WriteMessagePreamble(XmlDictionaryWriter writer) at System.ServiceModel.Channels.Message.OnWriteMessage(XmlDictionaryWriter writer) at System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message message, BufferManager bufferManager, Int32 initialOffset, Int32 maxSizeQuota) 
at System.ServiceModel.Channels.BinaryMessageEncoderFactory.BinaryMessageEncoder.WriteMessage(Message message, Int32 maxMessageSize, BufferManager bufferManager, Int32 messageOffset) 
at System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message message) 
at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout) 
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout) 
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) 
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) 
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) 
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) 
at IMyService.MyWcfMethod(int a, string b, double c) 
at MyClientClass.Process() 

Any ideas on where to search for the answer?

Answer 1

It sounds like it could be a threading issue - maybe the WCF client code is reusing the same serializer instance for multiple calls, and it's breaking stuff.

Can you try putting your call to client.MyWcfMethod() inside a lock block?

eg:

private static readonly object _wcfSync = new object();
...

  lock (_wcfSync)
  {
    client.MyWcfMethod(a, b, c);
  }