Multiple messages removed from MSMQ

Question

I have a certain service that runs over WCF and MSMQ. Everything works fine but we have this requirement which says that if something unexpected happens during processing of one job, it should not affect any other. The service is designed to be robust, and it is. We never had it crash in two yeas of operations however recently our customers have taking a liking to killing the service process and complaining that more than one job is lost.

To investigate I've added a timer which traces the contents of the queue every 200ms and found something unexpected. When processing requests, WCF removes more than one message from the queue. Here's a smartinspec trace of relevant events(the numbers in the center column are thread ids):

It always seems to keep one message buffered somewhere in the service process. I've been looking through WCF documentation and I haven't found any reference to this behavior Is there anything I can do to prevent this buffering or could you refer me to some relevant articles?

Here's the service class:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Single)]
 public class TheService : ITheService
 {
   private readonly ITraceFacade trace = TraceFactory.Resolve();//for tracing

   public ClientCodeWrapper clientCodeWrapper { get; set; }// a wrapper for libraries written by our client. it's instantiated and set in the OnStart method of the service host class

   public void ProcessJob(long jobId)
   {
      using (this.trace.ActivityScope(string.Format(CultureInfo.InvariantCulture, "The Service.ProcessJob({0})", jobId)))
       {        
         this.clientCodeWrapper.ProcessJob(jobId);        
       }
    }    
 }

and this is my service config

<system.serviceModel>
  <bindings>
    <netMsmqBinding>
      <binding name="MsmqBindingNonTransactionalNoSecurity" exactlyOnce="false">
        <security mode="None"/>
      </binding>
    </netMsmqBinding>
  </bindings>

  <services>
    <service name="TheService">
      <host>
        <baseAddresses>
          <add baseAddress="http://localhost:8080/TheServiceIn"   />           
        </baseAddresses>
      </host>           
      <endpoint address="net.msmq://localhost/private/TheServiceIn"  
              binding="netMsmqBinding"
              bindingConfiguration="MsmqBindingNonTransactionalNoSecurity" 
              contract="ITheService">         
        <identity>
          <dns value="localhost"/>
        </identity>
      </endpoint>       

      <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
    </service>
  </services>
</system.serviceModel>

Answer 1

What you're seeing seems just expected to me. It is true that WCF can read multiple messages of the queue for processing. Normally, this isn't a problem, and with transactional queues, this would definitely not be an issue (because each individual transaction would only commit after the message processing is complete, otherwise the transaction would abort and the message is returned to the queue).

So basically, you opted out of having messages being processed in a manner that avoids lost messages by disabling transactions and using non-transactional queues. Basically you told MSMQ and WCF that it's perfectly OK to occassionally lose messages.

Consider what would happen if your machine (or the MSMQservice) crashed: you wouldn't lose just one job; you'd lose all of them, since messages wouldn't be persisted.