How compilers handle the ABI correctly when calling a function by pointer in C/C++?

Question

How do the compilers know the ABI of the function a function pointer pointed to?

In computer software, an application binary interface (ABI) is an interface between two program modules.

Since in C/C++ a function pointer only specifies the API, there's no knowing what kind of ABI it's actually using, won't that be a problem to compilers, especially when they can't figure that out statically?

Does this mean that the programmer who use this kind of pointer needs to specify the calling convention by hand?

If that is the case, how to do this? And can anyone give me some link to the compiler documents on this?

Answer 1

If the function is using a different calling convention than the default for the platform you are on, then yes, you would need to specify the calling convention manually. Most platform try to use a single ABI, with a single or predictable set of calling conventions though, so any compiler would know how to call any function.

Though this is outside the scope of C++, and if you can or need to specify the calling convention, it would be done using a non-standard extension for the compiler you are using).

Answer 2

gcc and I suspect all major compilers solve this problem by having a function (and pointers to such functions) have different types for different calling conventions. Let's see:

__attribute__ ((noinline)) auto sum1(int a, int b) { return a + b; }
__attribute__ ((noinline)) auto sum2(int a, int b) __attribute__((fastcall));
auto sum2(int a, int b) { return a + b; }

auto test1(int a, int b) { return sum1(a, b); }
auto test2(int a, int b) { return sum2(a, b); }

sum1(int, int):
  mov eax, DWORD PTR [esp+8]
  add eax, DWORD PTR [esp+4]
  ret
sum2(int, int):
  lea eax, [ecx+edx]
  ret
test1(int, int):
  jmp sum1(int, int)
test2(int, int):
  mov edx, DWORD PTR [esp+8]
  mov ecx, DWORD PTR [esp+4]
  jmp sum2(int, int)

We can clearly see above that the two functions are called in different ways.

What happens when we throw pointers in the mix:

__attribute__ ((noinline)) auto call1(int a, int b, auto (*f)(int, int) -> int)
{
    return f(a, b);
}

__attribute__ ((noinline))
auto call2(int a, int b, auto (__attribute__((fastcall)) *f )(int, int) -> int  )
{
    return f(a, b);
}


auto test(int a, int b)
{
    call1(a, b, sum1);
    // call2(a, b, sum1); // compiler error

    // call1(a, b, sum2); // compiler error
    call2(a, b, sum2);
}

The compiler won't allow to convert a function pointer to a pointer to a function of a different calling convention.

error: invalid conversion from int (__attribute__((fastcall)) *)(int, int) to int (*)(int, int) [-fpermissive]

  call1(a, b, sum2); ^~~~ 

Play with it on godbolt