多重处理：分叉的缺点？

Question

We are experiencing an issue with Python Celery (which uses multiprocessing) where large periodic (scheduled) tasks consume massive amounts of memory for short bursts of time, but because the worker process lives through the life of the pool ( MAX_TASKS_PER_CHILD=None ), the memory is not garbage collected (ie. it is being "high-water" reserved).

(This problem is further worsened by Heroku, which sees a large, constant amount of memory allocated and turns it into swap, which decreases performance.)

We have found that by setting MAX_TASKS_PER_CHILD=1 , we fork a new process (Celery worker instance) after every task, and memory is properly garbage collected. Sweet!

However, there are plenty of articles that suggest the same solution, but I have not identified any downsides. What are the potential downsides of forking a new process after every task?

My guesses would be:
1. CPU overhead (but probably a tiny amount)
2. Potential errors when forking (but I can't find any documentation on this)

Answer 1

Aside from the obvious increase in CPU overhead from repeated forking (not a big deal if the workers do enough work per task), one possible downside would be if the parent process continues to grow in size. If so, it increases the size of all the child processes (which are forking a larger and larger parent). This wouldn't matter so much (presumably little of the memory will be written, and therefore little copying is required and actual memory use won't be a major issue), but IIRC, the Linux overcommit heuristics assume that the COW memory will eventually be copied, and you could invoke the OOM killer even if you're nowhere near actually exceeding the heuristic limit in terms of private pages.

On Python 3.4 and higher, you can avoid this issue by explicitly setting your multiprocessing start method to forkserver on program launch (before doing any work the workers don't rely on), which will fork workers from a separate server process that should not dramatically increase in size.