並行化 python for 循環

Question

我想使用帶有 2 個四核處理器的 Mac Pro 並行化以下 python 循環。

result_list = []
for a in a_range:
    for b in b_range:
        for c in c_range:
            result = call_fortran_program(a, b, c)
            result_list.append(result)

在我的搜索中，我遇到過像 Cython 和 GIL 這樣的術語，但我仍然不清楚如何繼續。

Answer 1

from itertools import product
from multiprocessing import Pool

with Pool(processes=4) as pool:  # assuming Python 3
    pool.starmap(print, product(range(2), range(3), range(4)))

Answer 2

嘗試使用ProcessPoolExecutor

通過創建多個進程，可以避免GIL鎖定。

網站上的示例：

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()

Answer 3

您可以使用異步。 （可以在 [此處][1] 找到文檔）。 它被用作多個 Python 異步框架的基礎，這些框架提供高性能的網絡和網絡服務器、數據庫連接庫、分布式任務隊列等。此外，它還具有高級和低級 API 以適應任何類型的問題。

import asyncio

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def your_function(argument):
    #code

現在這個 function 將在每次調用時並行運行，而無需讓主程序等待 state。您也可以使用它來並行化 for 循環。 當調用 for 循環時，雖然循環是順序的，但是一旦解釋器到達那里，每次迭代都會與主程序並行運行。

對於您的情況，您可以執行以下操作：

import asyncio
import time
from itertools import product


def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)                                   
    return wrapped
    
@background
def call_fortran_program(a, b, c):
    time.sleep(max(0,b-a+c)) # Added Sleep to better demonstrate parallelization
    print(f"function called for {a=}, {b=}, {c=}\n", end='')
    return (a, b, c)

a_range, b_range, c_range = range(1,4), range(1,4), range(1,4)

loop = asyncio.get_event_loop()          # Have a new event loop

looper = asyncio.gather(*[call_fortran_program(a, b, c) for a in a_range for b in b_range for c in c_range])                   # Run the loop
                             
result_list = loop.run_until_complete(looper) # Wait until finish

print(result_list)

這給出以下 output：

function called for a=1, b=1, c=1
function called for a=1, b=2, c=1
function called for a=1, b=1, c=2
function called for a=2, b=1, c=1
function called for a=1, b=3, c=1
function called for a=1, b=2, c=2
function called for a=1, b=1, c=3
function called for a=2, b=1, c=2
function called for a=1, b=3, c=2
function called for a=1, b=2, c=3
function called for a=2, b=1, c=3
function called for a=2, b=2, c=1
function called for a=3, b=1, c=1
function called for a=3, b=1, c=2
function called for a=3, b=2, c=1
function called for a=2, b=2, c=2
function called for a=2, b=3, c=1
function called for a=3, b=2, c=2
function called for a=3, b=1, c=3
function called for a=2, b=2, c=3
function called for a=1, b=3, c=3
function called for a=3, b=3, c=1
function called for a=2, b=3, c=2
function called for a=3, b=2, c=3
function called for a=3, b=3, c=2
function called for a=2, b=3, c=3
function called for a=3, b=3, c=3
[(1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 3, 1), (1, 3, 2), (1, 3, 3), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 3, 1), (2, 3, 2), (2, 3, 3), (3, 1, 1), (3, 1, 2), (3, 1, 3), (3, 2, 1), (3, 2, 2), (3, 2, 3), (3, 3, 1), (3, 3, 2), (3, 3, 3)]