[英]Garbage collector tries to collect shared memory object
I've got two Python scripts that both should do essentially the same thing: grab a large object in memory, then fork a bunch of children. 我有两个Python脚本在本质上都应该做同样的事情:在内存中抓取一个大对象,然后派生一堆子对象。 The first script uses bare
os.fork
: 第一个脚本使用裸
os.fork
:
import time
import signal
import os
import gc
gc.set_debug(gc.DEBUG_STATS)
class GracefulExit(Exception):
pass
def child(i):
def exit(sig, frame):
raise GracefulExit("{} out".format(i))
signal.signal(signal.SIGTERM, exit)
while True:
time.sleep(1)
if __name__ == '__main__':
workers = []
d = {}
for i in xrange(30000000):
d[i] = i
for i in range(5):
pid = os.fork()
if pid == 0:
child(i)
else:
print pid
workers.append(pid)
while True:
wpid, status = os.waitpid(-1, os.WNOHANG)
if wpid:
print wpid, status
time.sleep(1)
The second script uses multiprocessing
module. 第二个脚本使用
multiprocessing
模块。 I'm running both on Linux (Ubuntu 14.04), so it should use os.fork
under the hood too, as documentation states: 我都在Linux(Ubuntu 14.04)上运行,因此它也应该在
os.fork
使用os.fork
,如文档所述:
import multiprocessing
import time
import signal
import gc
gc.set_debug(gc.DEBUG_STATS)
class GracefulExit(Exception):
pass
def child(i):
def exit(sig, frame):
raise GracefulExit("{} out".format(i))
signal.signal(signal.SIGTERM, exit)
while True:
time.sleep(1)
if __name__ == '__main__':
workers = []
d = {}
for i in xrange(30000000):
d[i] = i
for i in range(5):
p = multiprocessing.Process(target=child, args=(i,))
p.start()
print p.pid
workers.append(p)
while True:
for worker in workers:
if not worker.is_alive():
worker.join()
time.sleep(1)
The difference between those two scripts is the following: when I kill a child (sending a SIGTERM), bare-fork script tries to garbagecollect the shared dictionary, despite the fact that it is still referenced by parent process and isn't actually copied into child's memory (because of copy-on-write) 这两个脚本之间的区别如下:当我杀死一个孩子(发送SIGTERM)时,裸叉脚本会尝试对共享字典进行垃圾回收,尽管事实上它仍被父进程引用并且实际上没有被复制到孩子的记忆(由于写时复制)
kill <pid>
Traceback (most recent call last):
File "test_mp_fork.py", line 33, in <module>
child(i)
File "test_mp_fork.py", line 19, in child
time.sleep(1)
File "test_mp_fork.py", line 15, in exit
raise GracefulExit("{} out".format(i))
__main__.GracefulExit: 3 out
gc: collecting generation 2...
gc: objects in each generation: 521 3156 0
gc: done, 0.0024s elapsed.
( perf record -e page-faults -g -p <pid>
output:) (
perf record -e page-faults -g -p <pid>
输出:)
+ 99,64% python python2.7 [.] PyInt_ClearFreeList
+ 0,15% python libc-2.19.so [.] vfprintf
+ 0,09% python python2.7 [.] 0x0000000000144e90
+ 0,06% python libc-2.19.so [.] strlen
+ 0,05% python python2.7 [.] PyArg_ParseTupleAndKeywords
+ 0,00% python python2.7 [.] PyEval_EvalFrameEx
+ 0,00% python python2.7 [.] Py_AddPendingCall
+ 0,00% python libpthread-2.19.so [.] sem_trywait
+ 0,00% python libpthread-2.19.so [.] __errno_location
While multiprocessing-based script does no such thing: 尽管基于多处理的脚本没有做到这一点:
kill <pid>
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "test_mp.py", line 19, in child
time.sleep(1)
File "test_mp.py", line 15, in exit
raise GracefulExit("{} out".format(i))
GracefulExit: 2 out
( perf record -e page-faults -g -p <pid>
output:) (
perf record -e page-faults -g -p <pid>
输出:)
+ 62,96% python python2.7 [.] 0x0000000000047a5b
+ 32,28% python python2.7 [.] PyString_Format
+ 2,65% python python2.7 [.] Py_BuildValue
+ 1,06% python python2.7 [.] PyEval_GetFrame
+ 0,53% python python2.7 [.] Py_AddPendingCall
+ 0,53% python libpthread-2.19.so [.] sem_trywait
I can also force the same behavior on multiprocessing-based script by explicitly calling gc.collect()
before raising GracefulExit
. 我还可以通过在引发
GracefulExit
之前显式调用gc.collect()
在基于多处理的脚本上强制执行相同的行为。 Curiously enough, the reverse is not true: calling gc.disable(); gc.set_threshold(0)
奇怪的是,事实并非如此:调用
gc.disable(); gc.set_threshold(0)
gc.disable(); gc.set_threshold(0)
in bare-fork script doesn't help to get rid of PyInt_ClearFreeList
calls. 裸机脚本中的
gc.disable(); gc.set_threshold(0)
不能摆脱PyInt_ClearFreeList
调用。
To the actual questions: 到实际问题:
Couple things 夫妻事
In python, multiple python processes means multiple interpreters with their own GIL, GC et al 在python中,多个python 进程意味着多个解释器具有自己的GIL,GC等
The d
dictionary is not passed in as an argument to the process, it is a globally shared variable. d
字典不作为进程的参数传递,它是全局共享变量。
The reason it gets collected is because each process thinks its the only one holding a reference to it which, strictly speaking, is true as it's a single globally shared object reference to the dictionary. 之所以收集它,是因为每个进程都认为它是唯一持有对其引用的引用,严格来说,这是正确的,因为它是对字典的单个全局共享对象引用。
When Python GC checks it, it checks the ref counter for that object. 当Python GC检查它时,它将检查该对象的ref计数器。 Since there is only the one shared reference, removing that would mean
ref count == 0
, so it gets collected. 由于只有一个共享引用,因此删除该引用将意味着
ref count == 0
,因此将其收集起来。
To resolve the issue, d
should be passed into each forked process, making each process hold its own reference to it. 要解决此问题,应将
d
传递到每个分支的进程中,使每个进程对其都有自己的引用。
Multiprocessing behaves differently because it uses os._exit
which doesn't call exit handler, which, apparently, involves garbage collection ( more on the topic ). 多重处理的行为有所不同,因为它使用
os._exit
而不调用退出处理程序,这显然涉及垃圾收集( 有关本主题的更多内容 )。 Explicitly calling os._exit
in bare-fork version of the script achieves the same result. 在脚本的裸叉版本中显式调用
os._exit
可获得相同的结果。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.