Fork系统调用失败后，rax中的返回值是多少？

Question

I know that calling fork() in C will return -1 if there is an error, but I'm wondering what the error return value is when you call sys_fork in assembly.

I might normally assume that it also returns -1 but I've dealt with sys_brk and the raw syscall in assembly returns something different from the C Brk() wrapper.

Does anyone know what the fork error return value would be in assembly?

(I'm doing 64 bit NASM assembly on Linux)

Answer 1

First and foremost note that the C library wrapper fork(2) invokes sys_clone and not sys_fork .

C library/kernel differences

Since version 2.3.3, rather than invoking the kernel's fork() system
call, the glibc fork() wrapper that is provided as part of the NPTL
threading implementation invokes clone(2) with flags that provide the
same effect as the traditional system call.

---

The introduction to the section 2 of the Linux manual explains how to interpret the return value of a system call in a general context:

RETURN VALUE

On error, most system calls return a negative error number (ie, the
negated value of one of the constants described in errno(3)). The C
library wrapper hides this detail from the caller: when a system call
returns a negative value, the wrapper copies the absolute value into
the errno variable, and returns -1 as the return value of the
wrapper.

So for most system calls, EAX/RAX holds -ESOMETHING on error, or a non-negative result on success. The libc wrappers decode this to implement the errno-setting and returning -1 behaviour described in the section 2 man pages, which primarily document the wrappers; "C library/kernel differences" details are sometimes found in the Notes section of Linux man pages.

It's important to note that this applies to most but not all system calls, as the first paragraph says. sys_fork is not special in this regard. A couple interesting special cases are getpriority as mentioned in the beginning of errno(3) ^{more below} , and mmap . (Valid pointers can have their high bit set, so distinguishing error from success requires other tricks, like checking the low bits since a successful mmap always returns a page-aligned address .) These ABI details are not documented in the man pages.

---

For the purpose of finding out if the sys_fork call was successful, testing for a negative value is enough:

test eax, eax
jl _error_handler              ;See Peter Cordes's comments below

I've included the part about the C library wrappers like fork(2) because it gives a practical method to find out the error numbers.
The value of errno , negated, are the possible error values the system call can return.

EAGAIN
ENOMEM
ENOSYS
ERESTARTNOINTR

In general, the C library wrapper can add, remove or transcode the return values before writing them into errno , so this is failable.

The definitive way to find out the possible return values is looking at the source.
For example, do_fork , invoked by sys_fork , can return EINVAL and EPERM in addition to the values listed above.
Other values are possible, I haven't dug into all the nested function calls.

sys_clone also invokes do_fork so I would assume that clone(2) can return all of the error numbers that fork(2) can.

---

Investigating the case of sys_getpriority mentioned above a comment came up

/*
 * Ugh. To avoid negative return values, "getpriority()" will
 * not return the normal nice-value, but a negated value that
 * has been offset by 20 (ie it returns 40..1 instead of -20..19)
 * to stay compatible.
 */

So it seems that Linux system calls always return negative values on an error, the C library wrappers than, in an attempt to normalise these values into errno , introduce an extra layer of complexity.

As we see from the mmap case, having the sign bit set doesn't always mean it's an error value. According to this answer (merit goes to Peter Cordes comment below), values in the range [-4095, -1] always mean error, but other negative values should not.