When does Linux x86-64 syscall clobber %r8, %r9 and %r10?

Question

I have just browsed the Linux kernel source tree and read the file tools/include/nolibc/nolibc.h .

I saw the syscall in this file uses %r8 , %r9 and %r10 in the clobber list.
Also there is a comment that says:

rcx and r8..r11 may be clobbered, others are preserved.

As far as I know, syscall only clobbers %rax , %rcx and %r11 (and memory).

Is there a real example of syscall that clobbers %r8 , %r9 and %r10 ?

Answer 1

Only 32-bit system calls (eg via int 0x80 ) in 64-bit mode step on those registers, along with R11. ( What happens if you use the 32-bit int 0x80 Linux ABI in 64-bit code? ).

syscall properly saves/restores all regs including R8, R9, and R10, so user-space using it can assume they keep their values, except the RAX return value. (The kernel's syscall entry point even saves RCX and R11, but at that point they've already been overwritten by the syscall instruction itself with the original RIP and before-masking RFLAGS value.)

---

Those, with R11, are the non-legacy registers that are call-clobbered in the function -calling convention, so compiler-generated code for C functions inside the kernel naturally preserves R12-R15, even if an asm entry point didn't save them.

Currently the 64-bit int 0x80 entry point just pushes 0 for the call-clobbered R8-R11 registers in the process-state struct that it will restore from before returning to user space, instead of the original register values. Perhaps earlier there were info leaks where it left them untouched after kernel code returned?

Answer 2

According to x86-64 ABI about syscall section A.2 AMD64 Linux Kernel Conventions, A.2.1 Calling Conventions [1]:

1. User-level applications use as integer registers for passing the sequence %rdi , %rsi , %rdx , %rcx , %r8 and %r9 . The kernel interface uses %rdi , %rsi , %rdx , %r10 , %r8 and %r9 .

2. A system-call is done via the syscall instruction. The kernel destroys registers %rcx and %r11 .

3. The number of the syscall has to be passed in register %rax .

4. System-calls are limited to six arguments, no argument is passed directly on the stack.

5. Returning from the syscall , register %rax contains the result of the system-call. A value in the range between -4095 and -1 indicates an error, it is -errno.

6. Only values of class INTEGER or class MEMORY are passed to the kernel.

From (2), (5) and (6), we can conclude that Linux x86-64 syscall clobbers %rax , %rcx and %r11 (and "memory" ).

Link: https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-psABI [1]