How do I parallelize for a for loop in Python like Matlab's parfor?

Question

I am new to Python and am coming from a Matlab-only coding background. I am trying to teach myself Python and am starting by rewriting one of my Matlab codes in Python. I have a bit of code where I am trying to accomplish the following: loop over a minimization problem where I minimize a function independently for each element of various input arrays while skipping over known nonfinite inputs. In Matlab, I just use parfor which is relatively straightforward.

parfor iter = (1:end_iter)
    if onlyfinite(iter) == 1
        tau_y(iter) = fminsearch(@(ty) ((s1(iter) - (ty / (p*g*ds1(iter)))) - (s2(iter) - (ty / (p*g*ds2(iter)))))^2,100000);
    end
end

I have achieved this in Python using numpy and scipy with the use of an ordinary for loop.

for idx in end_iter:
    if onlyfinite[idx] == 1:
        tau_y[idx] = optimize.fmin(lambda ty : ((s1[idx] - (ty / (p*g*ds1[idx]))) - (s2[idx] - (ty / (p*g*ds2[idx]))))**2,100000,disp=False)

But for the very large arrays I am working with, I need parallelization. I've tried to wrap my head around it but haven't made any progress. Any help to push me in the right direction is much appreciated!

Thanks in advance.

Answer 1

One approach would be to launch the optimizations as parallel tasks inside your for loop and waiting for them to finish afterwards:

from concurrent.futures import ThreadPoolExecutor

# create parallel executor
with ThreadPoolExecutor() as executor:
    threads = []
    for idx in end_iter:
        if onlyfinite[idx] == 1:
            # define function, arguments and keyword arguments
            fn = optimize.fmin
            args = [
                lambda ty: (
                    (s1[idx] - (ty / (p * g * ds1[idx])))
                    - (s2[idx] - (ty / (p * g * ds2[idx])))
                )
                ** 2,
                100000,
            ]
            kwargs = {
                "disp": False,
            }
        else:
            # dummies to keep idx order
            fn = lambda: None
            args = []
            kwargs = {}
        # start thread by submitting it to the executor
        threads.append(executor.submit(fn, *args, **kwargs))
# get results once they are finished
tau_y = [t.result() for t in threads]

Depending on your workload, it might be better to replace ThreadPoolExecutor with ProcessPoolExecutor, but this is a topic on its own .

Answer 2

You can look at this link, some basic are available about parallel programming in python...

https://joblib.readthedocs.io/en/latest/parallel.html