Python打开文件过多（子进程）

Question

I seem to be having an issue with Python when I run a script that creates a large number of sub processes. The sub process creation code looks similar to:

Code:

def execute(cmd, stdout=None, stderr=subprocess.STDOUT, cwd=None):
    proc = subprocess.Popen(cmd, shell=True, stdout=stdout, stderr=stderr, cwd=cwd)
    atexit.register(lambda: __kill_proc(proc))
    return proc

The error message I am receiving is:

OSError: [Errno 24] Too many open files

Once this error occurs, I am unable to create any further sub processes until kill the script and start it again. I am wondering if the following line could be responsible.

atexit.register(lambda: __kill_proc(proc))

Could it be that this line creates a reference to the sub process, resulting in a "file" remaining open until the script exits?

Answer 1

So it seems that the line:

atexit.register(lambda: __kill_proc(proc))

was indeed the culprit. This is probably because of the Popen reference being kept around so the process resources aren't free'd. When I removed that line the error went away. I have now changed the code as @Bakuriu suggested and am using the process' pid value rather than the Popen instance.

Answer 2

Firstly, run ulimit -a to find out how many the maximum open files are set in your Linux system.

Then edit the system configuration file /etc/security/limits.conf and add those code in the bottom.

* - nofile 204800

Then you can open more sub processes if you want.