How can I write in (GNU) C a proxy function to interface two different calling conventions?

Question

I'm writing an interpreter/compiler hybrid where the calling convention passes parameters on the CPU stack. Functions are simply pointers to machine code (like C function pointers) potentially generated at runtime. I need a proxy function to interface with the custom calling convention. I want to write as much as possible of this function in C , although necessarily some parts will have to be written in assembly. I will refer to this proxy function as apply .

I don't fully understand the semantics of GCC inline assembly and I would like to know if the following tentative implementation of a 1-ary apply function is correct, or where it goes wrong . In particular, I wonder about the integrity of the stack between the many __asm__ blocks: how does the compiler (GCC and clang in my case) interpret the stack pointer register being clobbered, and what are the consequences of that in the generated code? Does the compiler understand that I want to "own" the stack? Is the memory clobber necessary?

Through experimentation I found that clang with -fomit-frame-pointer correctly disables this optimization for a function when it sees the rsp register in a clobber list, since rsp is obviously not anymore a reliable way of addressing local variables on the stack. This is not true in GCC, and as a consequence it generates buggy code (this seems like a bug in GCC). So I guess this answers some of my questions. I can live with -fno-omit-frame-pointer , but it seems as if GCC doesn't consider the various implications of rsp being clobbered.

This is written for x86-64, although I am interested in eventually porting it to other architectures. We assume all registers are preserved across calls in the custom calling convention.

#define push(x) \
    __asm__ volatile ("pushq %0;" : : "g" (x) : "rsp", "memory")

#define pop(n) \
    __asm__ volatile ("addq %0, %%rsp;" : : "g" (n * 8) : "rsp", "memory")

#define call(f) \
    __asm__ volatile ("callq *%0;" : : "g" (f) : "cc", "memory")

void apply(void* f, void* x) {
  push(x);
  call(f);
  pop(1);
}

I think the -mno-red-zone flag is technically necessary to use the stack in the way I want. Is this correct?

---

The previous code assumes all registers are preserved across calls. But if there's a set of registers which aren't preserved, how should I reflect this in the code? I get the feeling that adding them to the call clobber list won't produce correct results because the registers may be pushed onto the top of the stack, shadowing the pushed x . If instead they are saved on a previously reserved area of the call frame, it may work. Is this the case? Can I rely on this behaviour? (Is it silly of me to hope so?)

Another option would be to manually preserve and restore these registers but I have a strong feeling this will only give the illusion of safety and break at some point.

Answer 1

I need a proxy function to interface with the custom calling convention. I want to write as much as possible of this function in C , although necessarily some parts will have to be written in assembly.

I'm sorry, this simply will not work. You must write the entire proxy function in assembly language.

More concretely -- I don't know about clang, but GCC assumes at a very basic level that nobody touches the stack pointer in inline assembly, ever. That doesn't mean it will error out -- it means it will blithely mis-optimize on the assumption that you didn't do that, even though you told it you did . This is not something that is likely ever to change; it's baked into the register allocator and all umpteen CPU back ends.

Now, the good news is, you may be able to persuade libffi to do what you want. It's got proxy functions which someone else has written in assembly language for you; if it fits your use case, it'll save you quite a bit of trouble.