ENOMEM from popen() for system(), while there is enough memory

Question

I have two relatively big applications(processes) running in embedded Linux on ARM with 3 RAM banks (in Linux cmdline: mem=128M mem=256M@0x90000000 mem=128M@0xA0000000). One application processes user commands, between which there may be request to run ordinary Linux shell command. This is implemented as:

if((fp=popen(UserCommand, "r")) == NULL) return(errno)); fgets(ReplyString, 128, fp); Res = pclose(fp);

The first line returns errno=12 - ENOMEM even for the simplest command like "pwd", although there is plenty of memory:

root@dm814x-evm:~# free total used free shared buffers Mem: 461472 38576 422896 0 152 Swap: 0 0 0 Total: 461472 38576 422896

As far as I understand there is more than 400MB of free space!
For the first test purpose I also cancelled the second process - ops!, the error has gone!!!
For the second test I run telnet and executed the command via it (while both processes were running) - no problem, works fine.

So, where is the catch?

Answer 1

If popen is implemented with fork() (as it currently is in glibc) rather than posix_spawn() or vfork() , then you need as much memory as what the parent is using for the operation to succeed. This allocation might fail if overcommit on your system is disabled, so to solve your problem, you should do one of the following:

1. fully enable overcommit on your system

    sudo sh -c 'echo 1>/proc/sys/vm/overcommit_memory' 

2. use a libc library that doesn't implement popen() with fork() ( musl is an example )

3. do what popen does yourself but express it in terms of pipe() and posix_spawn() , rather than the classical pipe() , dup2() , fork() , execve() combo.

( popen 1) creates a pipe 2) wraps one end of the pipe in a FILE and 3) creates a process and wires up the other end of the pipe to the processe's stdin or stdout , saving the process's pid in the FILE structure)

The problem with the classical fork() + execve() way of process creation is that if overcommit is disabled, fork() has to be pessimistic and assume the child might continue running the parent process, which would mean the child needs every page of the parent's memory. With overcommit enabled, the memory is borrowed and problems (the out-of-memory killer) only arise if the borrowed memory is accessed (which it won't be if the process will call execve() soon). With overcommit disabled, all of the memory has to be reserved, which is why it's likely to fail if the fork call is made from a larger process.