[英]How to implement multiprocessing IPC with multiple queues?
I'm new to multiprocessing and multithreading.我是多处理和多线程的新手。 For learning purpose, i'm trying to achieve IPC using queues.出于学习目的,我正在尝试使用队列来实现 IPC。
Code代码
from multiprocessing import Process, Queue, Lock
import math
def calculate_square(sq_q, sqrt_q):
itm = sq_q.get()
print(f"Calculating sq of: {itm}")
square = itm * itm
sqrt_q.put(square)
def calculate_sqroot(sqrt_q, result_q):
itm = sqrt_q.get()
print(f"Calculating sqrt of: {itm}")
sqrt = math.sqrt(itm)
result_q.put(sqrt)
sq_q = Queue()
sqrt_q = Queue()
result_q = Queue()
for i in range(5, 20):
sq_q.put(i)
p_sq = Process(target=calculate_square, args=(sq_q, sqrt_q))
p_sqrt = Process(target=calculate_sqroot, args=(sqrt_q, result_q))
p_sq.start()
p_sqrt.start()
p_sq.join()
p_sqrt.join()
while not result_q.empty():
print(result_q.get())
Explanation解释
Here i'm trying to run two of the functions with two different processes each of which calculates the square of the number and again calculate the square root of the number.在这里,我尝试使用两个不同的过程运行两个函数,每个过程计算数字的平方并再次计算数字的平方根。
Queues队列
Queue containing the initial number whose square root is to calculated.
sq_q: Queue containing the initial number whose square root is to calculated.
Queue containing the numbers whose square root has to be calculated
sqrt_q: Queue containing the numbers whose square root has to be calculated
Queue containing final result.
result_q: Queue containing final result.
Issue问题
Only the first item of the
sq_q
is consumed.仅消费sq_q
的第一项。
Output: Output:
5.0
I expect the output to be:
[5, 6, 7, 8, .. , 19]
我希望 output 为:[5, 6, 7, 8, .. , 19]
Please note that this is purely for learning purpose and i want to implement the IPC with multiple queues though it could have been possible to achieve with shared objects Locks and arrays.请注意,这纯粹是为了学习目的,我想用多个队列实现 IPC,尽管它可以通过共享对象锁和 arrays 来实现。
You only call the functions once, so only the first value 5 is taken, you then want to loop all the values in the queue.您只调用一次函数,因此只取第一个值 5,然后您想要循环队列中的所有值。
while not sq_q.empty():
itm = sq_q.get()
print(f"Calculating sq of: {itm}")
square = itm * itm
sqrt_q.put(square)
Same goes for the other function, but the condition here will be till result_q
is full (give a max size to the result queue for the condition to work), then the final result will have the values.其他 function 也是如此,但这里的条件将是直到result_q
已满(为结果队列提供最大大小以使条件起作用),然后最终结果将具有值。
while not result_q.full():
itm = sqrt_q.get()
print(f"Calculating sqrt of: {itm}")
sqrt = math.sqrt(itm)
result_q.put(sqrt)
Complete Code完整代码
import math
from multiprocessing import Process, Queue
def calculate_square(sq_q, sqrt_q):
while not sq_q.empty():
itm = sq_q.get()
print(f"Calculating sq of: {itm}")
square = itm * itm
sqrt_q.put(square)
def calculate_sqroot(sqrt_q, result_q):
while not result_q.full():
itm = sqrt_q.get()
print(f"Calculating sqrt of: {itm}")
sqrt = math.sqrt(itm)
result_q.put(sqrt)
if __name__ == "__main__":
sq_q = Queue()
sqrt_q = Queue()
result_q = Queue(5)
for i in range(5):
sq_q.put(i)
p_sq = Process(target=calculate_square, args=(sq_q, sqrt_q))
p_sqrt = Process(target=calculate_sqroot, args=(sqrt_q, result_q))
p_sq.start()
p_sqrt.start()
p_sq.join()
p_sqrt.join()
while not result_q.empty():
print(result_q.get())
Output Output
Calculating sq of: 0
Calculating sq of: 1
Calculating sq of: 2
Calculating sq of: 3
Calculating sq of: 4
Calculating sqrt of: 0
Calculating sqrt of: 1
Calculating sqrt of: 4
Calculating sqrt of: 9
Calculating sqrt of: 16
0.0
1.0
2.0
3.0
4.0
Edit:编辑:
As the calculate_sqroot
now depends on result_q
a delay is no longer needed.由于calculate_sqroot
现在依赖于result_q
,因此不再需要延迟。
Also we can use JoinableQueue which provides two of the most convenient method so that we can block the main process until the items in the queue are fully consumed.我们也可以使用JoinableQueue ,它提供了两种最方便的方法,这样我们就可以阻塞主进程,直到队列中的项目被完全消耗掉。
i.一世。 task_done()任务完成()
Indicate that a formerly enqueued task is complete.指示以前排队的任务已完成。 Used by queue consumers.由队列消费者使用。 For each
get()
used to fetch a task, a subsequent call totask_done()
tells the queue that the processing on the task is complete.对于用于获取任务的每个get()
,对task_done()
的后续调用会告诉队列该任务的处理已完成。
If a
join()
is currently blocking, it will resume when all items have been processed (meaning that atask_done()
call was received for every item that had beenput()
into the queue).如果join()
当前处于阻塞状态,它将在处理完所有项目后恢复(这意味着对于已将put()
放入队列的每个项目都收到了task_done()
调用)。
Raises a
ValueError
if called more times than there were items placed in the queue.如果调用的次数多于队列中放置的项目,则引发ValueError
。
ii. ii. join()加入()
Block until all items in the queue have been gotten and processed.阻塞直到队列中的所有项目都已被获取和处理。
The count of unfinished tasks goes up whenever an item is added to the queue.每当将项目添加到队列中时,未完成任务的计数就会增加。 The count goes down whenever a consumer calls
task_done()
to indicate that the item was retrieved and all work on it is complete.每当消费者调用task_done()
以指示该项目已被检索并且所有工作都已完成时,计数就会下降。 When the count of unfinished tasks drops to zero,join()
unblocks.当未完成任务的计数降至零时,join()
会解除阻塞。
Solution解决方案
from multiprocessing import Process, Lock
from multiprocessing import Queue
from multiprocessing import JoinableQueue
import math, time
def calculate_square(sq_q, sqrt_q):
while True:
itm = sq_q.get()
print(f"Calculating sq of: {itm}")
square = itm * itm
sqrt_q.put(square)
sq_q.task_done()
def calculate_sqroot(sqrt_q, result_q):
while True:
itm = sqrt_q.get() # this blocks the process unless there's a item to consume
print(f"Calculating sqrt of: {itm}")
sqrt = math.sqrt(itm)
result_q.put(sqrt)
sqrt_q.task_done()
items = [i for i in range(5, 20)]
sq_q = JoinableQueue()
sqrt_q = JoinableQueue()
result_q = JoinableQueue()
for i in items:
sq_q.put(i)
p_sq = Process(target=calculate_square, args=(sq_q, sqrt_q))
p_sqrt = Process(target=calculate_sqroot, args=(sqrt_q, result_q))
p_sq.start()
p_sqrt.start()
sq_q.join()
sqrt_q.join()
# result_q.join() no need to join this queue
while not result_q.empty():
print(result_q.get())
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.