防止多处理“失控的错误”

Question

The OS will not like it if you use multiprocessing and accidentally end up creating processes without limit .

Is there any simple solution that prevents this from happening (say, by limiting total number of processes, either in Python or in the OS)?

I use Windows, and it behaves really badly (requires hard reboot) when I make a mistake like that. So I'd love it if there's some code that I can wrap around / add to my application and prevent this from happening.

Answer 1

What you can do is create a short 'trip-wire' type module and import it as well as multiprocessing. The trip-wire module will raise an exception if it detects a multiprocessing infinite loop.

Mine looks like this:

#mp_guard.py
"""tracks invocation by creating an environment variable; if that
variable exists when next called a loop is in progress"""

import os

class Brick(Exception):
    def __init__(self):
        Exception.__init__(self, "Your machine just narrowly avoided becoming"
                                 " a brick!")

if 'MP_GUARD' in os.environ:
    raise Brick

os.environ['MP_GUARD'] = 'active'

And in the main .py file:

import mp_guard

Answer 2

On Linux, you can use the setrlimit(2) syscall (with RLIMIT_NPROC ) to limit the number of processes (eg to avoid fork bombs ). This syscall is interfaced thru the bash ulimit (or zsh limit ) builtin. Python has some binding to this syscall.

^{I have no idea if something similar exist under Windows.}

Answer 3

On windows you can create "a job", I'm not an expert in python, so I dont't know if theres is any binding for creating windows jobs. The Windows API function is CreateJobObject . A job object is (to some extent) equivalent of Unix "process group". You can apply certain limitations both on the job object as a whole and to each process separately (eg max processes in job). You could create a job object, assign number of processes limit to it and then assign your own process to the job object. What you may be looking for is CreateJobObject , SetInformationJobObject + JOBOBJECT_BASIC_LIMIT_INFORMATION + JOB_OBJECT_LIMIT_ACTIVE_PROCESS . Again: this is Windows API, I'm not sure if you have any 'bindings' to python related to these functions.