在python中使用多处理来划分for循环

Question

I want to create a graph and since there are 2600 nodes and I have to iterate through each node to create an edge, my for loop is running for almost 6 million times. Therefore, I am trying to do multiprocessing to make this faster. Now what I want, is to create 20 processes and divide the 6 million lines between them. I wrote this following code, but it doesn't work.

from igraph import *
from multiprocessing import Process    

def generate_edges(call_record_dict):
    for index, key in enumerate(call_record_dict):
        for index2, key2 in enumerate(call_record_dict):
            if(key!=key2):
                if(call_record_dict[key]==call_record_dict[key2]):
                    if(g.are_connected(index, index2) is False):
                        g.add_edges((index,index2))

def generate_graph(call_record_dict):
    g=Graph()
    g.add_vertices(len(call_record_dict))
    for i in range(20):
        p = Process(target=generate_edges, args=(call_record_dict))
        p.start()

I've tried using Pool.

pool = Pool(processes=20)
pool.map(generate_edges,call_record_dict)
pool.close()
pool.join()

This doesn't solve the problem either.

Answer 1

Try this:

from igraph import *
import  multiprocessing as mp

def generate_edges(call_record_dict):
    for index, key in enumerate(call_record_dict):
        for index2, key2 in enumerate(call_record_dict):
            if(key!=key2):
                if(call_record_dict[key]==call_record_dict[key2]):
                    return (index, index2)



def generate_graph(call_record_dict):
    g=Graph()
    g.add_vertices(len(call_record_dict))
    pool = mp.Pool(4)
    for index, ret in pool.map(generate_edges, call_record_dict):
        # ret = (index, index2)
        if(g.are_connected(ret) is False):
            g.add_edges((ret))

I dont have igraph and never used it so sorry this is not fully tested

Answer 2

If the values in the call_record_dict are hashable and are unique for each connection, you might try a different approach.

from collections import defaultdict
import itertools as it

matches = defaultdict(list)

for index, value in enumerate(call_record_dict.values()):
    matches[value].append(index)

for values in matches.values():
    for index1, index2 in it.combinations(values, 2):
        g.add_edge((index1, index2))
        g.add_edge((index2, index1))

Run time should be O(n), where n in the length of the dictionary, rather than O(n^2).