C++ Visual Studio Runtime Error

Question

Can anyone explain to me what this means?

"Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention."

Answer 1

When calling a function, the compiler has to push some arguments on the stack, or put them in some registers. The function body will change some memory location (or a register) to contain the return value. Then it will return to a block of code at a location stored 'somewhere' on the stack.

The calling convention specifies the concrete details: eg first push the return address, then push arguments (input or output) on the stack from left to right, then execute the function, pop the arguments off again, then pop the return address and jump to that location.

If the caller does this differently than the function expects, things can go wrong (return location is not at the expected stack position).

ESP is typically the register containing the address of the current stack frame. This register is used in combination with indexes to obtain arguments in the function body. When returning, the stack top is typically reset to ESP, and the processor jumps to the location at eg ESP+1.

Things that may cause this to happen:

• someone wrote over the values of the stack and changed the return location (eg buffer overflow)
• the caller had a different calling convention than the callee

Answer 2

Possibility #1 is your object got Aliased wrong. That means the compiler mistakenly casted an object incorrectly, most commonly as an optimization.

Possibility #2 your using a reference to an object that isn't valid.

Possibility #3 something totally unrelated broke your stack, likely a buffer overflow.

Without code sample It's hard to predict beyond these possibilities, but they cover 90% of what might be happening.

Answer 3

It means that the value in the processor ESP register was not restored correctly after a call to a function.

As has already been said function arguments are pushed on to the stack, and dumped after the function ends by adjusting the esp register. In certain calling conventions the esp register is adjusted after the function call returns by the calling code. In other conventions it is adjusted by the function that is called, just before it returns.

I have seen this error when a disagreement about who adjusts esp arises between the calling code and the code being called. Look carefully at how the function is declared, particularly if it is in a separate library or dll.

Answer 4

This is typically the case when casting a function pointer to shut up the compiler when calling a Windows API, such as DialogBox:

DialogBox(hInstance, MAKEINTRESOURCE(MY_DIALOG), hWnd, &dlgProc);

Windows is a bit odd in that all functions are defined with the stdcall calling-convention, contrary to cdecl, the default in C. Therefor all functions that should be passed to a Windows API should be defined with WINAPI (a define for stdcall):

INT_PTR WINAPI dlgProc(HWND, UINT, WPARAM, LPARAM);

If you miss that and instead define your dlgProc as:

INT_PTR dlgProc(HWND, UINT, WPARAM, LPARAM);

Your code won't compile and you might be sorely tempted to just shut the frikkin' compiler up:

DialogBox(hInstance, MAKEINTRESOURCE(MY_DIALOG), hWnd, (DLGPROC)&dlgProc); // be a DLGPROC already, dammit!!

Don't do that. The program will crash. The compiler is your friend. The C-style cast operator is not. The compiler tried to tell you this was going to happen. And it was right. Listen to it.