Python 中的 for 循环多处理或多线程

Question

我在 python 中有以下三个 for 循环（其中两个是嵌套的）。 API 请求应同时发送。 如何并行执行？

args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))

pool = ThreadPool(len(args))
# list of packets_received:
results = pool.starmap(my_function, args)
for result in results:
    # unpack the result tuple:
    packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted=result

#for sw1
for m in range (1,4,1):         
     print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)
                                    
#for sw2,sw3,sw4
for n in range (2,5,1):
    for m in range (2,5,1):
            print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)                           
#for sw5 to sw13
for n in range (5,14,1):
    for m in range (1,5,1):
        print(packets_recieved,packets_transmitted,bytes_recieved,bytes_transmitted)
    

Answer 1

查看apiString的三个实例并重写它们以使用更简洁的F-strings ，它们似乎都具有以下形式：

apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'

如果n为 1 并且m取值 1、2、3 和 4，即使第一个循环也适合这种格式。所以让我们修改my_function以采用两个 arguments， n和m ：

def my_function(n, m):
     apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'
    ... # //etc.
    return packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted

现在我们只需要安排使用以下 (n, m) 参数对同时调用它，这里指定为元组（暂时）：

args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))

并将它们放在一起：

from multiprocessing.pool import ThreadPool

def my_function(n, m):
    apiString = f'http://192.168.74.134:8181/restconf/operational/opendaylight-inventory:nodes/node/openflow:{n}/node-connector/openflow:{n}:{m}'
    ... # //etc.
    return packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted

args = []
args.extend((n, m) for n in (1,) for m in range(1, 4))
args.extend((n, m) for n in range(2, 5) for m in range(2, 5))
args.extend((n, m) for n in range(5, 14) for m in range(1, 5))

# Creating thread pool whose size is the number of requests we need to make:
# (but it could be smaller, for example 10, if you wanted less concurrency):
pool = ThreadPool(len(args))
# list of packets_received:
results = pool.starmap(my_function, args)
for result in results:
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    print(packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)

方法starmap将解包args中的每个元组，并将元组的元素作为单独的 arguments 传递给my_function 。

更新

我提供了 3 种方法来处理调用starmap返回的results 。 选择以下方法之一（但不是全部）：

# Option 1: Retrieve and print all N results corresponding to the N arguments in args
for result in results:
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    print(packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
    
# Option 2: Retrieve and print all N results in 3 separate loops:
it = iter(results) # create an iterator for results
for n in (1,):
    for m in range(1, 4):
        result = next(it) # get next result
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    # you can now also print the n and m value:
    print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
for n in (2, 5):
    for m in range(2, 5):
        result = next(it) # get next result
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    # you can now also print the n and m values:
    print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)
for n in (5, 14):
    for m in range(1, 5):
        result = next(it) # get next result
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    # you can now also print the n and m value:
    print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)

# Option 3: Using enumerate:
for idx, result in enumerate(results):
    arg = args[idx]
    # unpack the arg tuplet
    n, m = arg
    # unpack the result tuple:
    packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted = result
    # you can now also print the n and m values:
    print(n, m, packets_recieved, packets_transmitted, bytes_recieved, bytes_transmitted)