简体繁体 English

Python-类方法多进程安全吗？

[英]Python - Are class methods multiprocess safe?

原文 2010-11-17 22:47:36 6 2 python/ class/ iterator/ multiprocessing

I have a class that loops over some data files, processes them, and then writes new data back out. 我有一个类，它遍历一些数据文件，对其进行处理，然后将新数据写回。 The analysis of each file is completely independent of the others. 每个文件的分析完全独立于其他文件。 The class contains information needed by the analysis in its attributes, but the analysis does not need to change any attributes of the class. 该类在其属性中包含分析所需的信息，但分析不需要更改该类的任何属性。 Thus I can make the analysis of one data file a single method of my class. 因此，我可以使对一个数据文件的分析成为类的一种方法。 The analysis could in principle be done in parallel since each data file is independent. 由于每个数据文件都是独立的，因此分析原则上可以并行进行。 As an aside, I was considering making my class iterable. 顺便说一句，我正在考虑使我的课程变得可迭代。

Can I use the multiprocessing module to spawn processes that are methods of my class? 我可以使用多处理模块生成作为类方法的进程吗？ I need to use multiprocessing because I'm using third party code that has a really bad memory leak (fills up all 24Gb of memory after about 100 data files). 我需要使用多重处理，因为我使用的第三方代码的内存泄漏确实非常严重（大约100个数据文件后，所有24Gb内存都已填满）。

If not, how would you go about doing this? 如果没有，您将如何进行？ Would you just use a normal function called by my class (passing all the information I need as arguments) instead of a method? 您是否只使用类调用的普通函数（将我需要的所有信息作为参数传递）而不是方法？ How are arguments passed to functions in multiprocessing? 如何在多处理中将参数传递给函数？ Does it make a deep copy? 它会复制很深吗？

2 个解决方案

是的，如果您不更新需要在实例之间共享的类本身的数据，则在这种情况下，多处理是您的工具。

You're not mentioning your process using any external resources, so it should be fork()-safe. 您没有提及使用任何外部资源的过程，因此它应该是fork（）安全的。 Fork duplicates the memory and file descriptors, program state is identical in the parent and the child. Fork复制内存和文件描述符，在父级和子级中程序状态相同。 Unless you're using windows which can't fork, go for it. 除非您使用无法分叉的窗户，否则请继续。