python multiprocessing同时处理数字数组

Question

I have a list of number:

a=[1,2,3,4,5,.....2000]

I have to square each number and update the same array, but instead of writing a loop i want to do it using parallel processing.

So squaring each number in the array becomes a process in itself.

Expected output=[1,3,9,16,25,........]

How can i achieve this with python multiprocessing library?

Already tried to Use threading library but the code is not fast enough, plus Threading library is not using all the cores.

Answer 1

You can use Pool class from the multiprocessing module

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1, 2, 3]))

#prints [1, 4, 9]

Answer 2

In this numpy would be handy because it works on Matrix methods to calculate. Here is the piece of code that can serve the purpose. In case you want to parallel it you can use the Pool function as stated

import numpy as np
def Square(data):
    data_np = np.array(data) ** 2
    print (data_np)
Square([1, 2, 3])

Answer 3

You can try ProcessPoolExecutor in concurrent.futures module. Example code:

from time import time
from concurrent.futures import ProcessPoolExecutor


def gcd(pair):
    a, b = pair
    low = min(a, b)
    for i in range(low, 0, -1):
        if a % i == 0 and b % i == 0:
            return i


numbers = [(1963309, 2265973), (2030677, 3814172),
           (1551645,    2229620),   (2039045,   2020802)]
start = time()
results = list(map(gcd, numbers))
end = time()
print('1st Took %.3f seconds' % (end - start))
start = time()
pool = ProcessPoolExecutor(max_workers=2)
results = list(pool.map(gcd, numbers))
end = time()
print('2nd Took %.3f seconds' % (end - start))