[英]for loop Multiprocessing or Multithreading in Python
我在 python 中有以下三個 for 循環(其中兩個是嵌套的)。 API 請求應同時發送。 如何並行執行?
args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))
pool = ThreadPool(len(args))
# list of packets_received:
results = pool.starmap(my_function, args)
for result in results:
# unpack the result tuple:
packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted=result
#for sw1
for m in range (1,4,1):
print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)
#for sw2,sw3,sw4
for n in range (2,5,1):
for m in range (2,5,1):
print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)
#for sw5 to sw13
for n in range (5,14,1):
for m in range (1,5,1):
print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)
查看apiString
的三個實例並重寫它們以使用更簡潔的F-strings ,它們似乎都具有以下形式:
apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'
如果n
為 1 並且m
取值 1、2、3 和 4,即使第一個循環也適合這種格式。所以讓我們修改my_function
以采用兩個 arguments, n和m :
def my_function(n, m):
apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'
... # //etc.
return packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted
現在我們只需要安排使用以下 (n, m) 參數對同時調用它,這里指定為元組(暫時):
args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))
並將它們放在一起:
from multiprocessing.pool import ThreadPool
def my_function(n, m):
apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'
... # //etc.
return packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted
args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))
# Creating thread pool whose size is the number of requests we need to make:
# (but it could be smaller, for example 10, if you wanted less concurrency):
pool = ThreadPool(len(args))
# list of packets_received:
results = pool.starmap(my_function, args)
for result in results:
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
print(packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
方法starmap
將解包args
中的每個元組,並將元組的元素作為單獨的 arguments 傳遞給my_function
。
更新
我提供了 3 種方法來處理調用starmap
返回的results
。 選擇以下方法之一(但不是全部):
# Option 1: Retrieve and print all N results corresponding to the N arguments in args
for result in results:
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
print(packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
# Option 2: Retrieve and print all N results in 3 separate loops:
it = iter(results) # create an iterator for results
for n in (1,):
for m in range(1, 4):
result = next(it) # get next result
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
# you can now also print the n and m value:
print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
for n in (2, 5):
for m in range(2, 5):
result = next(it) # get next result
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
# you can now also print the n and m values:
print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
for n in (5, 14):
for m in range(1, 5):
result = next(it) # get next result
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
# you can now also print the n and m value:
print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
# Option 3: Using enumerate:
for idx, result in enumerate(results):
arg = args[idx]
# unpack the arg tuplet
n, m = arg
# unpack the result tuple:
packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
# you can now also print the n and m values:
print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.