[英]How to pass object from main process to child process using multiprocessing
I try to use Multiprocessing in a class.I use Multiprocessing.pipe() to pass the instance o from parent process to child process. 我尝试在类中使用Multiprocessing.I使用Multiprocessing.pipe()将实例o从父进程传递到子进程。
self.listofwritetags = self.collectwritetaglist()
self.progressbar['value'] = 20
self.frame.update_idletasks()
self.alldevice = alldevices_V3.AllDevices(self.comm_object)
self.progressbar['value'] = 40
self.frame.update_idletasks()
con1,con2 = multiprocessing.Pipe()
con1.send(self.alldevice)
con2.send(self.comm_object)
# Multithreading section
# self.callmotor1dprocess = thread_with_trace(target=self.callallmotor1d,args=(self.comm_object,self.alldevice))
self.callmotor1dprocess = multiprocessing.Process(target = self.callallmotor1d,args= (con1,con2))
self.listofthread.append(self.callmotor1dprocess)
self.button2.config(text="Initialized")
initial = "True"
self.progressbar.stop()
Now I call all Multiprocessing Initiate 现在,我将所有多处理都称为启动
def startprocess(self):
for item in self.listofthread:
item.start()
self.button3.config(text="started")
def stopprocess(self):
for item in self.listofthread:
item.kill()
This Code I call inside the class.Now Executed method I should call outside of the class. 我在类内部调用此代码。现在应该在类外部调用执行的方法。
def callallmotor1d(con1,con2):
comobject = con1.recv()
devices = con2.recv()
while True:
Allmotorprocessing.process(comobject, devices)
But I got the error which is very common:- Error message:- 但是我得到了非常常见的错误:-错误消息:-
Traceback (most recent call last): File "C:\\Users\\misu01\\AppData\\Local\\Programs\\Python\\Python37\\lib\\multiprocessing\\queues.py", line 236, in _feed obj = _ForkingPickler.dumps(obj) File "C:\\Users\\misu01\\AppData\\Local\\Programs\\Python\\Python37\\lib\\multiprocessing\\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects Traceback (most recent call last): File "C:\\Users\\misu01\\AppData\\Local\\Programs\\Python\\Python37\\lib\\multiprocessing\\queues.py", line 236, in _feed obj = _ForkingPickler.dumps(obj) File "C:\\Users\\misu01\\AppData\\Local\\Programs\\Python\\Python37\\lib\\multiprocessing\\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects 追溯(最近一次通话):文件“ C:\\ Users \\ misu01 \\ AppData \\ Local \\ Programs \\ Python \\ Python37 \\ lib \\ multiprocessing \\ queues.py”,第236行,位于_feed obj = _ForkingPickler.dumps(obj)文件中“ C:\\ Users \\ misu01 \\ AppData \\ Local \\ Programs \\ Python \\ Python37 \\ lib \\ multiprocessing \\ reduction.py”,第51行,在转储中cls(buf,protocol).dump(obj)TypeError:无法腌制_thread .lock对象回溯(最近一次调用):文件“ C:\\ Users \\ misu01 \\ AppData \\ Local \\ Programs \\ Python \\ Python37 \\ lib \\ multiprocessing \\ queues.py”,第236行,位于_feed obj = _ForkingPickler.dumps( obj)文件“ C:\\ Users \\ misu01 \\ AppData \\ Local \\ Programs \\ Python \\ Python37 \\ lib \\ multiprocessing \\ reduction.py”,第51行,位于转储cls(buf,protocol).dump(obj)TypeError:可以t泡菜_thread.lock对象
I don't know why thread.lock object is created. 我不知道为什么创建thread.lock对象。 To avoid this error I try to modify my Alldevices class and comm_object class like this:- 为避免此错误,我尝试像这样修改Alldevices类和comm_object类:
Here is my modification:- 这是我的修改:
class AllDevices: AllDevices类:
def __init__(self,comobject):
self.mylock = threading.Lock()
self.comobject = comobject
self.dfM1D = pd.read_excel(r'C:\OPCUA\Working_VF1_5.xls', sheet_name='Motor1D')
self.allmotor1dobjects = callallmotor1D_V3.Cal_AllMotor1D(self.dfM1D, self.comobject)
def __getstate__(self):
state = vars(self).copy()
# Remove the unpicklable entries.
del state['mylock']
return state
def __setstate__(self, state):
# Restore instance attributes.
vars(self).update(state)
Here is the comobject class. 这是comobject类。
class General():
def __init__(self):
self.client = Communication()
self.mylock = threading.Lock()
self.sta_con_plc = self.client.opc_client_connect()
self.readgeneral = ReadGeneral(self.client.PLC)
self.writegeneral = WriteGeneral(self.client.PLC)
def __getstate__(self):
state = vars(self).copy()
# Remove the unpicklable entries.
del state['mylock']
return state
def __setstate__(self, state):
# Restore instance attributes.
vars(self).update(state)
But still I got an error. 但是我还是有一个错误。
Is this my implementation is correct? 这是我的实现对吗?
self.allmotor1dobjects = callallmotor1D_V2.Cal_AllMotor1D(self.dfM1D, self.comobject,self.logger)
Here self.allmotor1dobjects is also class instances. 这里self.allmotor1dobjects也是类实例。
like:- 喜欢:-
self.client = Communication()
self.readgeneral = ReadGeneral(self.client.PLC)
self.writegeneral = WriteGeneral(self.client.PLC)
These are also class instances. 这些也是类实例。
I never used thread.lock any of these two classes. 我从未使用过thread.lock这两个类中的任何一个。 I don't know how it is created. 我不知道它是如何创建的。
As per the docs suggested https://docs.python.org/3/library/pickle.html#pickling-class-instances If I use getstate and setstate , it should remove this error. 根据文档建议https://docs.python.org/3/library/pickle.html#pickling-class-instances如果我使用getstate和setstate ,它应该消除此错误。
In my case it doesn't work. 就我而言,它不起作用。
How can I remove this error. 如何删除此错误。
any help in this regard will be highly appreciated 在这方面的任何帮助将不胜感激
You can pass most classes through a pipe, but not if the instances have attributes that are unpicklable types. 您可以通过管道传递大多数类,但是如果实例具有不可拾取类型的属性,则不能。 In this case, the instances of alldevice
(the instances of which are stored in some collection you stored as self.devices
) have, directly or indirectly, a threading.Lock
attribute (the error message says _thread.lock
because under all the abstraction, that's the actually class that threading.Lock
returns an instance of). 在这种情况下, alldevice
的实例(其实例存储在您存储为self.devices
某个集合中)具有直接或间接的threading.Lock
属性(错误消息为_thread.lock
因为在所有抽象下,那是threading.Lock
返回实例的实际类。
threading.Lock
isn't picklable (because thread locks only make sense within a given process; even if you recreated them in another process, they wouldn't actually provide any sort of synchronization between the processes). threading.Lock
是不可选的(因为线程锁仅在给定进程中才有意义;即使您在另一个进程中重新创建了它们,它们实际上也不会在进程之间提供任何形式的同步)。
If the alldevice
class is under your control, you have a few options: 如果alldevice
类在您的控制之下,则有几种选择:
threading.Lock
if possible (easiest solution, but assumes synchronization isn't required, or that synchronization could be done by a shared global lock) 删除每个实例的threading.Lock
如果可能的话,请锁定(最简单的解决方案,但是假设不需要同步,或者可以通过共享的全局锁来完成同步) multiprocessing.Lock()
instead; 如果需要同步,并且必须跨进程运行,但是不必按实例运行,并且您使用的是类似UNIX的系统(Linux,BSD),则可以使用共享的全局multiprocessing.Lock()
代替; it will be inherited when the Process
call triggers a fork
当Process
调用触发fork
时,它将被继承 If synchronization is required, and you must have a per-instance Lock
(and it's okay for the lock to operate across processes), you can replace threading.Lock
with a pickle-friendly multiprocessing.Manager
's Lock
. 如果需要同步,并且必须具有每个实例的Lock
(并且该锁可以跨进程运行是可以的),则可以使用对pickle友好的multiprocessing.Manager
的Lock
替换threading.Lock
。 You'd need to make a common multiprocessing.Manager()
instance somewhere (eg globally, or as a class attribute of alldevice
), then use Lock
from that manager instead of from threading
. 您需要在某个地方(例如,全局地或作为alldevice
的类属性multiprocessing.Manager()
创建一个通用的multiprocessing.Manager()
实例,然后使用该管理器中的Lock
而不是threading
。 Simple example: 简单的例子:
import multiprocessing class alldevice: MANAGER = multiprocessing.Manager() # Shared manager for all alldevice instances def __init__(self, ... other args here ...): self.mylock = self.MANAGER.Lock() # Make picklable lock # ... rest of initialization ...
When multiprocessing
isn't involved, this will be slower than a threading.Lock
, as it will require IPC to lock and unlock the lock (the Manager
's Lock
instances are actually a proxy object that communicates with the Manager
on whatever process it's actually running in), and it will lock across processes (probably for the best if you're locking access to actual hardware that can't be used concurrently from multiple processes), but it's relatively simple. 当不涉及multiprocessing
,这将比threading.Lock
慢,因为它将要求IPC锁定和解锁该锁( Manager
的Lock
实例实际上是一个代理对象,可以在其实际进行的任何进程上与Manager
进行通信运行),它将跨进程锁定(如果您要锁定对不能从多个进程中同时使用的实际硬件的访问,则可能是最好的选择),但这相对简单。
If synchronization is required, but only within a process, not across processes, you can take control of the pickling process to avoid trying to pickle the threading.Lock
, and instead recreate it when it's unpickled on the other side. 如果需要同步,而仅在一个进程内而不是跨进程,则可以控制腌制过程,以避免尝试腌制threading.Lock
,而是在另一侧未腌制时重新创建它。 Just explicitly implement the pickle support methods to avoid pickling the Lock
, and force it to be recreated on the other side. 只需显式实现pickle支持方法即可避免对Lock
进行腌制,并强制在另一侧重新创建它。 Example: 例:
import copy class alldevice: def __init__(self, ... other args here ...): self.mylock = threading.Lock() # Use regular lock # ... rest of initialization ... def __getstate__(self): state = vars(self).copy() # Make copy of instance dict del state['mylock'] # Remove lock attribute return state def __setstate__(self, state): vars(self).update(state) self.mylock = threading.Lock() # Make new lock on other side # If you ever make copies within the same thread, you may want to define # __deepcopy__ so the local copy process doesn't make a new lock: def __deepcopy__(self, memo): # Make new empty instance without invoking __init__ newself = self.__class__.__new__(self.__class__) # Individually deepcopy attributes *except* mylock, which we alias for name, value in vars(self).items(): # Cascading deepcopy for all other attributes if name != 'mylock': value = copy.deepcopy(value, memo) setattr(newself, name, value) return newself
The __deepcopy__
override is only needed if you want the copy to continue sharing the lock; 仅当您希望副本继续共享锁时才需要__deepcopy__
替代; otherwise, if a deep copy should behave as an entirely independent instance, you can omit it, and you'll end up with an unrelated lock in the copy. 否则,如果深层副本应表现为完全独立的实例,则可以将其省略,最终在副本中将获得不相关的锁定。
If you don't have control of the alldevice
class, but can identify the problematic attribute, your only option is to register a copyreg
handler for alldevice
to do the same basic thing as option #4, which would look something like this: 如果您没有对alldevice
类的控制权,但是可以识别有问题的属性,则唯一的选择是为alldevice
注册一个copyreg
处理程序,以执行与选项#4相同的基本操作,如下所示:
import copyreg
def unpickle_alldevice(state):
self = alldevice.__new__(alldevice) # Make empty alldevice
vars(self).update(state) # Update with provided state
self.mylock = threading.Lock() # Make fresh lock
return self
def pickle_alldevice(ad):
state = vars(ad).copy() # Make shallow copy of instance dict
del state['mylock'] # Remove lock attribute
return unpickle_alldevice, (state,) # Return __reduce__ style info for reconstruction
# Register alternate pickler for alldevice
copyreg.pickle(alldevice, pickle_alldevice)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.