Common Dialogs on Windows Server 2008 R2: crash in GetOpenFileName

Question

We have a sizable application, written in C++, running on Windows Server 2003 and Windows Server 2008 R2. It uses the GetOpenFileName API to bring up the Open File dialog, for selecting a video file.

We're seeing that on Windows Server 2008 R2, we occasionally get a crash in the Open File dialog. The main GUI thread is waiting for a worker thread, and the worker thread is getting an access violation exception. The stack trace looks like this:

linkinfo.dll!_IsValidLinkInfo@4()   
shell32.dll!LinkInfo_LoadFromStream()  + 0x7d bytes 
shell32.dll!CShellLink::_LoadFromStream()  + 0x14b bytes    
shell32.dll!CShellLink::Initialize()  + 0x1a bytes  
shell32.dll!InitializeFileHandlerWithStream()  + 0xc4 bytes 
shell32.dll!CFileSysItemString::HandlerCreateInstance()  + 0x13b bytes  
shell32.dll!CFileSysItemString::LoadHandler()  - 0x9f1d bytes   
shell32.dll!CFSFolder::_CreatePerInstanceDefExtIcon()  + 0x9b bytes 
shell32.dll!CFSFolder::_CreateDefExtIcon()  + 0xb6 bytes    
shell32.dll!CFSFolder::s_GetExtractIcon()  + 0x1e bytes 
shell32.dll!CFSFolder::_BindHandler()  - 0xd759 bytes   
shell32.dll!CFSFolder::GetThumbnailHandler()  + 0x51 bytes  
shell32.dll!_CreateThumbnailHandler()  + 0x61 bytes 
shell32.dll!CShellItem::BindToHandler()  - 0x20a93 bytes    
shell32.dll!_GetExtractIconW@16()  + 0x63 bytes 
shell32.dll!_GetILIndexFromItem()  + 0x5f bytes 
shell32.dll!_SHGetIconIndexFromPIDL@20()  - 0x3857d bytes   
shell32.dll!CFSFolder::GetIconOf()  + 0xa57c3 bytes 
shell32.dll!_SHGetIconIndexFromPIDL@20()  + 0x24 bytes  
shell32.dll!_SHIconIndexFromPIDL@16()  + 0x3d bytes 
shell32.dll!CRegFolder::GetIconOf()  + 0x10a892 bytes   
shell32.dll!_SHGetIconIndexFromPIDL@20()  + 0x24 bytes  
shell32.dll!_SHIconIndexFromPIDL@16()  + 0x3d bytes 
explorerframe.dll!CNscIconTask::_Extract()  + 0x1f bytes    
explorerframe.dll!CNscOverlayTask::InternalResumeRT()  + 0x31 bytes 
explorerframe.dll!CRunnableTask::Run()  + 0xa2 bytes    
shell32.dll!CShellTask::TT_Run()  + 0x5b bytes  
shell32.dll!CShellTaskThread::ThreadProc()  + 0x99 bytes    
shell32.dll!CShellTaskThread::s_ThreadProc()  + 0x1b bytes  
shlwapi.dll!_ExecuteWorkItemThreadProc@4()  + 0xe bytes 
ntdll.dll!_RtlpTpWorkCallback@8()  + 0xdf bytes 
ntdll.dll!_TppWorkerThread@4()  - 0x1185 bytes  
kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes    
ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes   
ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes    

It appears that a number of other people have run into similar problems: MSDN thread from 2011 .

"I have the same situation that opening the common dialog (8R2) the application crashes. Sometimes it helps to re-start the application and try it again..sometimes not. Since this happens for a couple of applications from various vendors, it's more likely an issue of the common dialog."

"I want to chime in here that we are experiencing the same problem with a 2008 R2 machine. It can be any program (and I've seen it once in Notepad.exe). When you look at the event viewer, you see that the crash happens in different modules that are called by the common dialog. It happens in 32-bit programs and 64-bit. It does not happen 100% of the time -- but rather about 50%. We have not seen any pattern to when and why it happens."

Our hypothesis is that the Open File dialog is trying to get some thumbnail information out of the video file, but that the video decoder is crashing.

Has anyone else run into this problem? If so, were you able to find out what the root cause was? Do you know anything about why IsValidLinkInfo would get an access violation?

Are there any workarounds that we could apply here? We're planning to try removing the file association for this particular file type (.ts). Is there any way to tell the Open File dialog not to create thumbnails?

Answer 1

Update:

We found another thread on ServerFault reporting a similar problem ("Typically it crashes when you click File/Open or File/Save, but not every time"). Another user (PG) was able to resolve the problem by excluding the application from Data Execution Prevention, on Microsoft's recommendation.

We checked the Data Execution Prevention setting and found that it was set to "Turn on DEP for all programs and services except those I select." After changing it to "Turn on DEP for essential Windows programs and services only", the problem no longer occurs.