How can my WCF service recover from unavailable message queue?

Question

I have a WCF service that receives messages from the Microsoft Message Queue ( netMsmqBinding ).

I want my service to recover if the message queue is unavailable. My code should fail to open the service, but then try again after a delay.

I have code to recognize the error when the queue is unavailable:

static bool ExceptionIsBecauseMsmqNotStarted(TypeInitializationException ex)
{
    MsmqException msmqException = ex.InnerException as MsmqException;
    return ((msmqException != null) && msmqException.HResult == (unchecked((int)(0xc00e000b))));
}

So this should be straightforward: I call ServiceHost.Open() , catch this exception, wait for a second or two, then repeat until my Open call is successful.

The problem is, if this exception gets thrown once, it continues to be thrown. The message queue might have become available, but my running process is in a bad state and I continue to get the TypeInitializationException until I shut down my process and restart it.

Is there a way around this problem? Can I make WCF forgive the queue and genuinely try to listen to it again?

---

Here is my service opening code:

public async void Start()
{
    try
    {
        _log.Debug("Starting the data warehouse service");
        while(!_cancellationTokenSource.IsCancellationRequested)
        {
            try
            {
                _serviceHost = new ServiceHost(_dataWarehouseWriter);
                _serviceHost.Open();
                return;
            }
            catch (TypeInitializationException ex)
            {
                _serviceHost.Abort();
                if(!ExceptionIsBecauseMsmqNotStarted(ex))
                {
                    throw;
                }
            }
            await Task.Delay(1000, _cancellationTokenSource.Token);
        }
    }
    catch (Exception ex)
    {
        _log.Error("Failed to start the service host", ex);
    }
}

And here is the stack information. The first time it is thrown the stack trace of the inner exception is:

at System.ServiceModel.Channels.MsmqQueue.GetMsmqInformation(Version& version, Boolean& activeDirectoryEnabled)

at System.ServiceModel.Channels.Msmq..cctor()

And the top entries of the outer exception stack:

at System.ServiceModel.Channels.MsmqChannelListenerBase`1.get_TransportManagerTable()

at System.ServiceModel.Channels.TransportManagerContainer..ctor(TransportChannelListener listener)

Answer 1

Microsoft have made the source code to WCF visible, so now we can work out exactly what's going on.

The bad news: WCF is implemented in such a way that if the initial call to ServiceModel.Start() triggers a queueing error there is no way to recover.

The WCF framework includes an internal class called MsmqQueue . This class has a static constructor . The static constructor invokes GetMsmqInformation , which can throw an exception.

Reading the C# Programming Guide on static constructors:

If a static constructor throws an exception, the runtime will not invoke it a second time, and the type will remain uninitialized for the lifetime of the application domain in which your program is running.

There is a programming lesson here: Don't put exception throwing code in a static constructor!

The obvious solution lies outside of the code. When I create my hosting service, I could add a service dependency on the message queue service. However, I would rather fix this problem with code then configuration.

Another solution is to manually check that the queue is available using non-WCF code.

The method System.Messaging.MessageQueue.Exists returns false if the message queue service is unavailable. Knowing this, the following works:

private const string KNOWN_QUEUE_PATH = @".\Private$\datawarehouse";

private static string GetMessageQueuePath()
{
    // We can improve this by extracting the queue path from the configuration file
    return KNOWN_QUEUE_PATH;
}

public async void Start()
{
    try
    {
        _log.Debug("Starting the data warehouse service");
        string queuePath = GetMessageQueuePath();
        while(!_cancellationTokenSource.IsCancellationRequested)
        {
            if (!(System.Messaging.MessageQueue.Exists(queuePath)))
            {
                _log.Warn($"Unable to find the queue {queuePath}. Will try again shortly");
                await Task.Delay(60000, _cancellationTokenSource.Token);
            }
            else
            {
                _serviceHost = new ServiceHost(_dataWarehouseWriter);
                _serviceHost.Open();
                return;
            }
        }
    }
    catch(System.OperationCanceledException)
    {
        _log.Debug("The service start operation was cancelled");
    }
    catch (Exception ex)
    {
        _log.Error("Failed to start the service host", ex);
    }
}