x64 asm how to set a function pointer to a _cdecl C function and call it?

Question

I'm trying to do something pretty basic in x64 asm:

1. Have an asm function which take a function pointer and sets this in a variable. This function is called from C code.

2. Have another asm function which calls the function pointer if not null, this function pointer is also a C function (as as set by the function in 1).

Here is what I have so far for the C side of things:

extern "C" void _asm_set_func_ptr(void* ptr);

void _cdecl c_call_back()
{

}

void init()
{
    _asm_set_func_ptr(c_call_back);
}

And the asm side:

.DATA

g_pFuncPtr QWORD 0

.CODE             ;Indicates the start of a code segment.

_asm_set_func_ptr PROC fPtr:QWORD
    mov     [rsp+qword ptr 8], rcx
    mov     rax, [rsp+qword ptr 8]
    mov     g_pFuncPtr, rax
    ret
_asm_set_func_ptr ENDP 

_asm_func PROC

push RBX
push RBP
push RDI
push RSI
push RSP
push R12
push R13
push R14
push R15

CMP g_pFuncPtr, 0
JE SkipCall
    MOV RAX, [ g_pFuncPtr ];
    CALL RAX;
SkipCall:

pop RBX
pop RBP
pop RDI
pop RSI
pop RSP
pop R12
pop R13
pop R14
pop R15
ret

_asm_func ENDP 

But it seems I damage the stack after calling _asm_set_func_ptr(), also I'm not sure if how I call g_pFuncPtr in _asm_func is correct? What is wrong with my code? I'm building this with VS2013 MASM64.

Answer 1

First, you generally need to pop the registers in the reverse order in which you push them, ie:
push RBX , push RBP ... push R15 --> pop R15 ... pop RSI , pop RBX , ret . This will definitely break the caller of _asm_func .

---

Next you should look at the Windows x64 calling convention what all is necessary to make proper function calls. It is very important to get all the requirements right, otherwise things can break and even very late in some else's code, which is not the greatest thing to debug.

For example, you don't need to save all registers. If the callback function destroys them, it will save and restore them itself. So no pushing and popping is necessary there, RAX can be invalidated anyway, no argument is being passed in it.

But then note this part:

In the Microsoft x64 calling convention, it's the caller's responsibility to allocate 32 bytes of "shadow space" on the stack right before calling the function (regardless of the actual number of parameters used), and to pop the stack after the call.

So you should do SUB ESP, 32 before your code, then ADD ESP, 32 before the RET .

There is also the requirement for " stack aligned on 16 bytes ", but you don't currently need to address that, because "8 bytes of return address + 32 bytes of shadow space + 8 bytes of next return address" is aligned on 16 bytes.

Additionally, the Windows x64 ABI has also strict requirements on exception handling and correct unwinding. As Raymond pointed out in the comment, because your function is not a leaf one (calls other functions), you need to provide a proper prologue and epilogue instead -- see here .

---

The temporary saving of RCX at the beginning of _asm_set_func_ptr is unnecessary.

Otherwise I don't see any problems there, though.

---

Finally, semicolons ; are not needed at end of lines in assembler files.

Answer 2

You're pushing a lot of registers before checking g_pFuncPtr, but you're not popping them back off the stack if it's not been set. If you push something onto the stack & then don't make the call and don't pop them back, your stack will fill up fast.

You MUST pop registers in the opposite order of pushing them, or you'll get back the wrong registers.

Last, don't waste time & CPU cycles pushing them at all unless you have something to do with them:

    CMP g_pFuncPtr, 0
    JE SkipCall

    PUSH RBX
    PUSH RBP
    PUSH RDI
    PUSH RSI
    PUSH RSP
    PUSH R12
    PUSH R13
    PUSH R14
    PUSH R15
    MOV RAX, [ g_pFuncPtr ];
    CALL RAX;
    POP R15
    POP R14
    POP R13
    POP R12
    POP RSP
    POP RSI
    POP RDI
    POP RBP
    POP RBX
SkipCall:
    ret

... and please - please... do read up on setting up stack frames and managing stack frames inside calls. C calls and ASM calls handle stack frames very differently from each other.