Preventing blocking dialogs/message boxes/hanging GUI from non-interactive processes on Windows?

Question

We are developing C++ apps (lots of MFC) with Visual Studio 2005 on Windows.

From time to time it happens that our nightly builds and/or unit tests hang because some part of some app or helper tool opens a message box in a corner case that is hit by the build.

Since the automated stuff is run (by a Windows Service) without any desktop session attached, obviously no-one can confirm - or even read - the GUI messages.

Is there any way to have Windows prevent apps from opening dialogs? Or maybe a tool that watches a service session that auto-kills any app that opens a dialog box?

I'm thinking that most cases where apps display unexpected popup-messages, it will end up calling one of the MessageBox* functions from user32.dll and it might be just possible to "magically" have these functions fail for a certain login-session? (Just a wild idea.)

Obviously the "right" fix is to have stuff not opening any dialogs, but with 3rd party tools it ain't always possible and with our tools it would be nicer to have a failing unit test that tells me the test "illegally" opened a message box than have a hanging unit test.

(Side notes: We're using Boost.Test for our unit tests and FinalBuilder for our automatic build scripts.)

_{Note: Removed original tags [continuous-integration build-automation automated-tests] and rephrased question to be more process centric.}

Answer 1

We use AutoIt to automatically dismiss dialogs in our commercial run-as-windows-service application. The concept is described and some sample scripts are available here: http://www.coretechnologies.com/products/AlwaysUp/AutoIt/

Note that some of the AutoIt functions don't work properly in Session 0 (eg WinActivate) but you can usually find alternatives. Be sure to test in Session 0!

Answer 2

You can load a DLL in each of the processes which puts a hook onto MessageBoxA and MessageBoxW. You can either do this manually or via the Detours library. You can then either have it return straight away without calling the real function or you could even implement some form of logging to notify your CI of the error.