Python : How to optimize calculations?

Question

I'm making some text-mining from a corpus of words, and I'm having a textfile output with 3000 lines like this :

dns 11 11 [2, 355, 706, 1063, 3139, 3219, 3471, 3472, 3473, 4384, 4444]

xhtml 8 11 [1651, 2208, 2815, 3487, 3517, 4480, 4481, 4504]

javascript 18 18 [49, 50, 175, 176, 355, 706, 1063, 1502, 1651, 2208, 2280, 2815, 3297, 4068, 4236, 4480, 4481, 4504]

There is the word, the number of lines where it've appeared, the number of total appearances, and the n° of these lines.

I'm trying to calculate The Chi-squared Value, and that textfile is the input for my code below :

measure = nltk.collocations.BigramAssocMeasures()

dicto = {} 
for i in lines :
    tokens = nltk.wordpunct_tokenize(i)
    m = tokens[0]       #m is the word
    list_i = tokens[4:]
    list_i.pop()
    for x in list_i :
        if x ==',':
            ind = list_i.index(x)
            list_i.pop(ind)
    dicto[m]=list_i #for each word i create a dictionnary with the n° of lines

#for each word I calculate the Chi-squared with every other word 
#and my problem is starting right here i think
#The "for" loop and the z = .....


for word1 in dicto :
    x=dicto[word1]
    vector = []

    for word2 in dicto :    
        y=dicto[word2]
        z=[val for val in x if val in y]

        #Contingency Matrix
        m11 = cpt-(len(x)+len(y)-len(z))
        m12 = len(x)-len(z)
        m21 = len(y)-len(z)
        m22 = len(z)

        n_ii =m11
        n_ix =m11+m21
        n_xi =m11+m12
        n_xx =m11+m12+m21+m22 

        Chi_squared = measure.chi_sq(n_ii, (n_ix, n_xi), n_xx)

        #I compare with the minimum value to check independancy between words
        if Chi_squared >3.841 :
            vector.append([word1, word2 , round(Chi_square,3))

    #The correlations calculated
    #I sort my vector in a descending way
    final=sorted(vector, key=lambda vector: vector[2],reverse = True)

    print word1
    #I take the 4 best scores
    for i in final[:4]:
        print i,

My problem is that the calcul is taking to much time (I'm talking about Hours !!) Is there anything that I can change ? anything that I do to improve my code ? Any other Python structures ? any ideas ?

Answer 1

There are a few opportunities for speedup, but my first concern is vector . Where is it initialized? In the code posted, it gets n^2 entries and sorted n times! That seems unintentional. Should it be cleared? Should final be outside the loop?

final=sorted(vector, key=lambda vector: vector[2],reverse = True)

is functional, but has ugly scoping, better is:

final=sorted(vector, key=lambda entry: entry[2], reverse=True)

In general, to solve timing issues consider using a profiler .

Answer 2

First, if for every word you have unique line numbers, use sets instead of lists: finding set intersection is much faster than the intersection of lists (especially if lists are not ordered).

Second, precompute list lengths - now you compute it twice for every single inner cycle step.

And third - use numpy for this kind of computation.