When CDialog.DoModal() function fails to create dialog box?

Question

MSDN said, for the CDialog.DoModal() function, the return value is –1 if the function could not create the dialog box. It does not say in which case it could fail to create the dialog box.

A quick investigation into MFC source code shows if the (LPCDLGTEMPLATE)LockResource(hDialogTemplate) returns NULL, the DoModal function could return -1.

Since my problem cannot be reproduced stably in our stress test, I cannot debug the program to find the root cause. Did anybody here meet similar problem?

Answer 1

DoModal() relies on the CreateDialogIndirect() API function. The documentation for its cousin DialogBox() states that function can fail under the following circumstances:

• an invalid parameter value,
• the system class was registered by a different module,
• the WH_CBT hook is installed and returns a failure code,
• one of the controls in the dialog template is not registered, or its window procedure fails on WM_CREATE or WM_NCCREATE .

I personally never encountered the first three reasons, but I was bitten once by the fourth because my dialog box contained an ActiveX control that was not registered on the machine. Maybe you're experiencing the same issue.

Answer 2

I found the root cause comes from the exhausted GDI objects. Our software exists handle leaks. The max number of GDI objects can be set in registry. The value is 16000 in our Windows XP. I wrote a program to create lots of UI controls and not to free them. By this way I can simulate the GDI handle exhausting situation. As a result, the same problem is reproduced again.

HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Windows\\GDIProcessHandleQuota

Ref: http://msdn.microsoft.com/en-us/library/windows/desktop/ms724291%28v=vs.85%29.aspx

So is it the final conclusion that when the GDI objects are exhausted, some UI controls such as CDialog, would be failed to create?