[英]Python multiprocessing: RuntimeError: “Queue objects should only be shared between processes through inheritance”
I am aware of multiprocessing.Manager() and how it can be used to create shared objects. 我知道multiprocessing.Manager()以及它如何用于创建共享对象。 In particular, queues which can be shared among workers.
特别是,可以在工人之间共享的队列。 There is this question , this question , and this question .
有这个问题 ,这个问题 ,还有这个问题 。
However, these links don't mention why we can use inheritance for sharing between processes. 但是,这些链接没有提到为什么我们可以使用继承来在进程之间共享。 As I understand, a queue can still only be copied in this case.
据我了解,在这种情况下仍然只能复制队列。
The Queue
implementation in python relies on a system pipe
to transmit the data from one process to another and some semaphores
to protect the read and write on this pipe
. python中的
Queue
实现依赖于系统pipe
将数据从一个进程传输到另一个进程以及一些semaphores
来保护此pipe
上的读写。
The pipe
is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints. pipe
在进程中作为打开文件处理,并且由于操作系统限制,只能在生成时与子进程共享。
The semaphores
are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python. 对于早期版本的python,
semaphores
也被视为只应在产卵时共享的文件,至少在基于UNIX的系统中。
As these 2 sub objects cannot be shared in general, the Queue
cannot be pickled and sent to a subprocess once it has been started. 由于这两个子对象通常不能共享,因此一旦启动,就无法对
Queue
进行pickle并将其发送到子进程。
However, for some OS and recent version of python, it is possible to share the Connection
and to create sharable Semaphore
. 但是,对于某些操作系统和最新版本的python,可以共享
Connection
并创建可共享的Semaphore
。 Thus, you could in theory create your own Queue
that can be shared between processes. 因此,您可以在理论上创建可以在进程之间共享的自己的
Queue
。 But it involves a lot of hacks and might not be very secured. 但它涉及很多黑客,可能不是很安全。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.