用Python为table1的每个值读取table2的最快方法？

Question

I want to compute a matching or a marriage. I have, say, a men basket in table2, and women in table1. For each row of table1, I want to evaluate a distance for that particular individual idx1 with all rows in table2. Then I select the argmin idx2, and save somewhere that idx1 is matched with idx2. Algorithm is not finished because I want to remove idx2 from the men basket (table2) before going to next idx1. The distance is a function of variables from table1 and table2, typically, score_str = '(table2[age] - table1[age][idx1])^2 + (table2[diploma] - table1[diploma][idx1])^2' ( table1[varname][idx1] becomes temp[varname] in the code below)

Because, I use pandas, I wrote the following code, but it take 15 seconds to match around 2000 men and 2000 women. I'm not sure the use of pandas is an advantage here. I may have to change. Computing time is important in that case as I'll match much much bigger databases (around 2 millions).

Edit: first comments are right it's a quadratic algorithm so it will take time anyway and the size = 2,000,000 will surely still a dream. An other stage will be to split the big dataset in smaller chunks (but that will be done with an economist point of view). The faster is the algorithm, the biggest the chunks can be, so it's still important for me to improve computing time.

Thanks for any help.

import pandas as pd
import pdb
import numpy as np

size = 5000 
score_str = "(table2['age']-temp['age'])**2 +  5*(table2['diploma']-temp['diploma'])"

table2 = pd.DataFrame(np.random.randn(size, 2), columns=['age','diploma'])
table1 = pd.DataFrame(np.random.randn(size, 2), columns=['age','diploma'])

match = pd.Series(0, index=table1.index)
index2 = pd.Series(True, index=table2.index)  
k_max = min(len(table2), len(table1))
def matching():
    for k in xrange(k_max):   
        temp = table1.iloc[k] 
        score = eval(score_str)[index2]
        idx2 = score.idxmax()
        match.iloc[k] = idx2 # print( k, 0, index2)
        index2[idx2] = False

    return match

matching()

Edit : rather than the idea of RussW, I've translated my code from pandas to numpy. It's the first small step to a lower-level language, isn't it ? That way my simulation is for time faster. With n=2,000,000 the calculus lasts seven hours. In my world (microeconomics) it's look like a reasonable time.

def run_time_np(n):
    table2 = np.random.randint(0,100, [n,2])
    table1 = np.random.randint(0,100, [n,2])
    idx2 = np.array([np.arange(n)])
    table2 = np.concatenate((table2, idx2.T), axis=1)

    match = np.empty(n, dtype=int)
    k_max = min(len(table2), len(table1))
    score_str = "(table2[:,0]-temp[0])**2 +  5*(table2[:,1]-temp[1])"
    k_max = min(len(table2), len(table1))
    start = time.clock()
    for k in xrange(k_max):   
        temp = table1[k]
        score = eval(score_str)
        idx = score.argmax()
        idx2 = table2[score.argmax(),2]
        match[k] = idx2 
        table2 = np.delete(table2, idx, 0)
    print 'taille: ',n,' ; temps de calcul: ', time.clock()-start
    return match

Answer 1

You should certainly use a profiler to see where the code is spending time. You'll be able to see if panda is slowing down. Also as far as I understand your algorithm, it is quadratic O(n^2). I don't think you'll be able to run it in a reasonable time for a 2 million size table.

Answer 2

For every individual in one table, you'll be comparing that individual with every individual in the other table with a function (age difference)^2 + (diploma difference)^2 .

One idea to reduce the number of operations is to use groups/buckets to first find a minimal set of other individuals, and then compare with that minimal set using the same function to find a match.

Take a table and make 2 new tables, age_groups and dip_groups . In age_groups you would have age buckets (which can be done with a dict ) with keys such as (20, 25) -> (minage, maxage) . The same for dip_groups.

Then you algorithm would look like this: (pseudocode)

for individual in table1:
    age, diploma = individual
    for age_bucket, dip_bucket in iterate_buckets(age, diploma):
        matches = age_bucket.intersection(dip_bucket)
        if matches:
            match = get_best_match(matches, age, diploma)
            all_matches.append((individual, match))
            remove_individual(age_groups, match)
            remove_individual(dip_groups, match)

The main thing would be the iterate_buckets() and get_best_match() functions.

age_groups = [(18, 20), (21, 23), (24, 26), ... ]
dip_groups = [(1, 2), (3, 4), (5, 6) ... ]
group_combinations = [(ag, dg) for ag in age_groups for dp in dip_groups]

def iter_key(age_grp, dip_group, age, dip):
    age_av = sum(age_grp) / 2.0
    dip_av = sum(dip_grp) / 2.0
    return pow(age - age_av, 2) + pow(dip - dip_av, 2)

def iterate_buckets(age, dip):
    combs = sorted(group_combinations, key=lambda grp: iter_key(*grp, age, dip))
    for c in combs:
        yield c

def match_key(indiv1, indiv2):
    age1, dip1 = indiv1
    age2, dip2 = indiv2
    return pow(age1 - age2, 2) + pow(dip1 - dip2, 2)

def get_best_match(matches, age, dip):
    sorted_matches = sorted(key=match_key, zip(matches, [(age, dip)] * len(matches)))
    return sorted_matches[0]

Just an idea, I'm not 100% sure it would be faster or that it would produce the same desired result.