[英]Python Multiprocessing: pool.map vs using queues
I am trying to use the multiprocessing
package for Python
. 我正在尝试使用
Python
的multiprocessing
包。 In looking at tutorials, the clearest and most straightforward technique seems to be using pool.map
, which allows the user to easily name the number of processes and pass pool.map
a function and a list of values for that function to distribute across the CPUs. 在查看教程时,最清晰,最直接的技术似乎是使用
pool.map
,它允许用户轻松命名进程数pool.map
传递一个函数和一个值列表,以便在CPU之间分配。 The other technique that I have come across is using queues to manage a pool of workers. 我遇到的另一种技术是使用队列来管理工作池。 This answer does an excellent job explaining the difference between
pool.map
, pool.apply
, and pool.apply_async
, but what are the advantages and disadvantages of using pool.map
versus using queues like in this example ? 这个答案很好地解释了
pool.map
, pool.apply
和pool.apply_async
之间的区别,但是使用pool.map
与使用这个例子中的队列有什么优缺点?
The pool.map
technique is a "subset" of the technique with queues. pool.map
技术是具有队列的技术的“子集”。 That is, without having pool.map
you can easily implement it using Pool
and Queue
. 也就是说,如果没有
pool.map
您可以使用Pool
和Queue
轻松实现它。 That said, using queues gives you much more flexibility in controlling your pool processes, ie you can make it so that particular types of messages are read only once per processes' lifetime, control the pool processes' shutdown behaviour, etc. 也就是说,使用队列可以更灵活地控制池进程,也就是说,您可以使每个进程的生命周期只读取特定类型的消息一次,控制池进程的关闭行为等。
If you're really looking for the "clearest and most straightforward technique", using concurrent.futures.ProcessPoolExecutor
is probably the easiest way. 如果您真的在寻找“最清晰,最直接的技术”,那么使用
concurrent.futures.ProcessPoolExecutor
可能是最简单的方法。 It has a map
method as well as some other primitives that make it very usable. 它有一个
map
方法以及一些使它非常实用的其他原语。 It is also compatible with Queue
s. 它也与
Queue
兼容。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.