For one-way WCF service with MSMQ 3.0 binding: What is correct behavior for ServiceHost.Faulted event handling?

Question

Scenario 1 : In Faulted event I am restarting service which means recreates ServiceHost.

I have implemented IErrorHandler to handle poison messages and after moving poison messages to poison queue it calls RestartService method.

Now when message is poisoned first Faulted event is called which calls RestartService but after opening ServiceHost successfully very soon it get Faulted again but here I have handled to stoped restarting it further.

During this time IErrorHandler.HandleError is called on separated thread twice and for one it was not able to find message to move to poison queue and second call moved message to poison queue, at the end of each call to IErrorHandler.HandleError if exception was MsmqPoisonMessageException it will restarts service.

Below is sequence of methods getting called:

ProcessMessage – message throws exception every time e.g. DB server is down
    ServiceHost.Faulted 
        RestartService
            ServiceHost.Faulted
                RestartService – skipped as still first call to RestartService not finished
    IErrorHandler.HandleError - Message not found to move to poison queue
        RestartService
    IErrorHandler.HandleError - Message moved to poison queue
        RestartService – skipped as a call to RestartService by first call to IErrorHandler.HandleError is not yet finished

Scenario 2: In Faulted event I am NOT restarting service.

Below is sequence of methods getting called:

ProcessMessage – message throws exception every time e.g. DB server is down
    ServiceHost.Faulted
    IErrorHandler.HandleError - Message moved to poison queue
        RestartService 

Now my worry is if I follow scenario 2 which is more clean and performing but if service faulted by any other reason than MsmqPoisonMessageException service will be stopped responding.

What solution do you suggest in this case?

I am using MSMQ 3.0 on Windows XP and Windows Server 2003.

Answer 1

For MSMQ 3 we saw that restarting the service in FaultHandler is not a good idea, as it may lead to reprocessing of messages before the message can be moved to poison queue.

If the service is restarted before handling the poison message (which will happen as it will be faulted -> raising the faulted event before raising the poisonMessageException) the new service host might start processing that message and the poison handling cannot find the poisoned message then.

The Problem I see with this is: what to do if the WCF service has an exception and faults?