[英]Python multiprocessing: synchronizing file-like object
我正在尝试制作像 object 这样的文件,该文件旨在在测试期间分配给 sys.stdout/sys.stderr 以提供确定性 output。 它并不意味着快速,只是可靠。 到目前为止我所拥有的几乎可以工作,但我需要一些帮助来摆脱最后几个极端情况错误。
这是我目前的实现。
try:
from cStringIO import StringIO
except ImportError:
from StringIO import StringIO
from os import getpid
class MultiProcessFile(object):
"""
helper for testing multiprocessing
multiprocessing poses a problem for doctests, since the strategy
of replacing sys.stdout/stderr with file-like objects then
inspecting the results won't work: the child processes will
write to the objects, but the data will not be reflected
in the parent doctest-ing process.
The solution is to create file-like objects which will interact with
multiprocessing in a more desirable way.
All processes can write to this object, but only the creator can read.
This allows the testing system to see a unified picture of I/O.
"""
def __init__(self):
# per advice at:
# http://docs.python.org/library/multiprocessing.html#all-platforms
from multiprocessing import Queue
self.__master = getpid()
self.__queue = Queue()
self.__buffer = StringIO()
self.softspace = 0
def buffer(self):
if getpid() != self.__master:
return
from Queue import Empty
from collections import defaultdict
cache = defaultdict(str)
while True:
try:
pid, data = self.__queue.get_nowait()
except Empty:
break
cache[pid] += data
for pid in sorted(cache):
self.__buffer.write( '%s wrote: %r\n' % (pid, cache[pid]) )
def write(self, data):
self.__queue.put((getpid(), data))
def __iter__(self):
"getattr doesn't work for iter()"
self.buffer()
return self.__buffer
def getvalue(self):
self.buffer()
return self.__buffer.getvalue()
def flush(self):
"meaningless"
pass
...和一个快速测试脚本:
#!/usr/bin/python2.6
from multiprocessing import Process
from mpfile import MultiProcessFile
def printer(msg):
print msg
processes = []
for i in range(20):
processes.append( Process(target=printer, args=(i,), name='printer') )
print 'START'
import sys
buffer = MultiProcessFile()
sys.stdout = buffer
for p in processes:
p.start()
for p in processes:
p.join()
for i in range(20):
print i,
print
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
print
print 'DONE'
print
buffer.buffer()
print buffer.getvalue()
这在 95% 的情况下都能完美运行,但它存在三个极端情况问题。 我必须在一个快速的 while 循环中运行测试脚本来重现这些。
在最坏的情况下(几率:七千万分之一),output 看起来像这样:
START
DONE
302 wrote: '19\n'
32731 wrote: '0 1 2 3 4 5 6 7 8 '
32732 wrote: '0\n'
32734 wrote: '1\n'
32735 wrote: '2\n'
32736 wrote: '3\n'
32737 wrote: '4\n'
32738 wrote: '5\n'
32743 wrote: '6\n'
32744 wrote: '7\n'
32745 wrote: '8\n'
32749 wrote: '9\n'
32751 wrote: '10\n'
32752 wrote: '11\n'
32753 wrote: '12\n'
32754 wrote: '13\n'
32756 wrote: '14\n'
32757 wrote: '15\n'
32759 wrote: '16\n'
32760 wrote: '17\n'
32761 wrote: '18\n'
Exception in thread QueueFeederThread (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
File "/usr/lib/python2.6/threading.py", line 484, in run
File "/usr/lib/python2.6/multiprocessing/queues.py", line 233, in _feed
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
在 python2.7 中,例外情况略有不同:
Exception in thread QueueFeederThread (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 505, in run
File "/usr/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
<type 'exceptions.IOError'>: [Errno 32] Broken pipe
我如何摆脱这些边缘情况?
解决方案分为两部分。 我已经成功运行了 20 万次测试程序,output 没有任何变化。
简单的部分是使用 multiprocessing.current_process()._identity 对消息进行排序。 这不是已发布的 API 的一部分,但它是每个进程的唯一确定性标识符。 这解决了 PID 环绕并给出 output 错误排序的问题。
解决方案的另一部分是使用 multiprocessing.Manager().Queue() 而不是 multiprocessing.Queue。 这解决了上面的问题 #2,因为管理器存在于一个单独的进程中,因此在使用来自拥有进程的队列时避免了一些糟糕的特殊情况。 #3 是固定的,因为队列已完全耗尽,并且在 python 开始关闭并关闭标准输入之前,馈线线程自然死亡。
我遇到的 Python 2.7 的multiprocessing
错误比 Python 2.6 少得多。 话虽如此,我用来避免“ Exception in thread QueueFeederThread
的异常”问题的解决方案是在使用Queue
的每个进程中暂时sleep
,可能为 0.01 秒。 确实,使用sleep
是不可取的,甚至是不可靠的,但观察到指定的持续时间在实践中对我来说效果很好。 你也可以试试0.1s。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.