在 Python 2.7 中并行化 for 循环

Question

I'm very new to Python (and coding in general) and I need help parallising the code below. I looked around and found some packages (eg. Multiprocessing & JobLib) which could be useful.

However, I have trouble using it in my example. My code makes an outputfile, and updates it doing the loop(s). Therefore is it not directly paralisable, so I think I need to make smaller files. After this, I could merge the files together.

I'm unable to find a way to do this, could someone be so kind and give me a decent start?

I appreciate any help, A code newbie

Code:

def delta(graph,n,t,nx,OutExt):
    fout_=open(OutExt+'Delta'+str(t)+'.txt','w')
    temp=nx.Graph(graph)
    for u in range(0,n):
        #print "stamp: "+str(t)+" node: "+str(u)
        for v in range(u+1,n):
            #print str(u)+"\t"+str(v)
            Stat = dict()
            temp.add_edge(u,v)
            MineDeltaGraphletTransitionsFromDynamicNetwork(graph,temp,Stat,u,v)
            for a in Stat:
                for b in Stat[a]:
                    fout_.write(str(t)+"\t"+str(u)+"\t"+str(v)+"\t"+str(a)+"\t"+str(b)+"\t"+str(Stat[a][b])+"\n")
            if not graph.has_edge(u,v):
                temp.remove_edge(u,v)
    del temp
    fout_.close()

Answer 1

As a start, find the part of the code that you want to be able to execute in parallel with something (perhaps with other invocations of that very same function). Then, figure out how to make this code not share mutable state with anything else.

Mutable state is the enemy of parallel execution. If two pieces of code are executing in parallel and share mutable state, you don't know what the outcome will be (and the outcome will be different each time you run the program). This is becaues you don't know what order the code from the parallel executions will run in. Perhaps the first will mutate something and then the second one will compute something. Or perhaps the second one will compute something and then the first one will mutate it. Who knows? There are solutions to that problem but they involve fine-grained locking and careful reasoning about what can change and when.

After you have an algorithm with a core that doesn't share mutable state, factor it into a separate function (turning locals into parameters).

Finally, use something like the threading (if your computations are primarily in CPython extension modules with good GIL behavior) or multiprocessing (otherwise) modules to execute the algorithm core function (which you have abstracted out) at some level of parallelism.

The particular code example you've shared is a challenge because you use the NetworkX library and a lot of shared mutable state. Each iteration of your loop depends on the results of the previous, apparently. This is not obviously something you can parallelize. However, perhaps if you think about your goals more abstractly you will be able to think of a way to do it (remember, the key is to be able to expressive your algorithm without using shared mutable state).

Your function is called delta . Perhaps you can split your graph into sub-graphs and compute the deltas of each (which are now no longer shared ) in parallel.

If the code within your outermost loop is concurrent safe (I don't know if it is or not), you could rewrite it like this for parallel execution:

from multiprocessing import Pool

def do_one_step(nx, graph, n, t, OutExt, u):
    # Create a separate output file for this set of results.
    name = "{}Delta{}-{}.txt".format(OutExt, t, u)
    fout_ = open(name, 'w')
    temp = nx.Graph(graph)

    for v in range(u+1,n):
        Stat = dict()
        temp.add_edge(u,v)
        MineDeltaGraphletTransitionsFromDynamicNetwork(graph,temp,Stat,u,v)
        for a in Stat:
            for b in Stat[a]:
                fout_.write(str(t)+"\t"+str(u)+"\t"+str(v)+"\t"+str(a)+"\t"+str(b)+"\t"+str(Stat[a][b])+"\n")
        if not graph.has_edge(u,v):
            temp.remove_edge(u,v)
    fout_.close()

def delta(graph,n,t,nx,OutExt):
    pool = Pool()
    pool.map(
        partial(
            do_one_step,
            nx,
            graph,
            n,
            t,
            OutExt,
        ),
        range(0,n),
    )

This supposes that all of the arguments can be serialized across processes (required for any argument you pass to a function you call with multiprocessing ). I suspect that nx and graph may be problems but I don't know what they are.

And again, this assumes it's actually correct to concurrently execute the inner loop.

Answer 2

Best use pool.map. Here an example that shows what you need to do. Here a simple example of how multiprocessing works with pool:

Single threaded, basic function:

def f(x):
    return x*x

if __name__ == '__main__':
     print(map(f, [1, 2, 3]))

>> [1, 4, 9]

Using multiple processors:

from multiprocessing import Pool 

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(3) # 3 parallel pools
    print(p.map(f, [1, 2, 3]))

Using 1 processor

from multiprocessing.pool import ThreadPool as Pool 

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(3) # 3 parallel pools
    print(p.map(f, [1, 2, 3]))

When you use map you can easily get a list back from the results of your function.