花费太长时间来运行python脚本

Question

The script is basically measuring the relationship between students based on the book they have borrowed (code). So I've built a tree for different type of books using ete2 package. Now I'm trying to write a piece of code that takes the data from the tree and a csv file and does some data analysis through the function relationship.The csv file contains more than 50,000 rows. The problem is that it takes to long to run the code (around 7 days), while it uses only 10 to 20% of my computer CPU and memory.

Here is an example of the csv file I've used:

ID Code    Count 
1    A1...   6
1    A2...   5
2    A....   4
2    D....   1
2    A1...   2
3    D....   5
3    D1...   3
3    D2...   5

Here is the code:

from ete2 import Tree
import pandas as pd
import numpy as np
from __future__ import division
import math


data= pd.read_csv('data.csv', names=['ID','Code', 'Count'])
codes_list= list (set(data['Code']))
total_codes= data.shape[0]
students_list= list (set(data['ID']))



####################################


# generate the tree
t = Tree (".....;", format =0)
for i in codes_list:
    if '....' in i:
        node = t.search_nodes(name = '.....')
        node[0].add_child(name= i)
for i in codes_list:
    if '...' in i and '....' not in i:
        if i[0]+'....' in codes_list:
            node = t.search_nodes(name = i[0]+'....')
            node[0].add_child(name= i)
        else:
            node = t.search_nodes(name = '.....')
            node[0].add_child(name= i)

# save the tree in a file 
t.write( outfile= file_path + "Codes_tree.nh", format =3)
return t.get_ascii(show_internal=True)

####################################

def relationship(code1,code2):

    code1_ancestors= t.search_nodes(name=code1)[0].get_ancestors()
    code2_ancestors=t.search_nodes(name=code2)[0].get_ancestors(
    common_ancestors = []
    for a1 in code1_ancestors:
        for a2 in code2_ancestors:
            if a1==a2:
                common_ancestors.append(a1)
    IC_values = []
    for ca in common_ancestors:
        code_descendants=[]
        for gd in ca.get_descendants():
            code_descendants.append(gd.name)
        code_descendants.append(ca)
        frequency= 0
        for k in code_descendants:
                frequency= frequency + code_count.Count[k]

        IC = - (math.log (frequency / float (total_codes)))
        IC_values.append (IC)

    IC_max= max(IC_values)
    return IC_max

##################

relationship_matrix = pd.DataFrame(index=[students_list], columns=[students_list])
for student in students_list:
p1= list (self.data.Code[data.ID==student])
for student1 in students_list:
    p2= list data.Code[data.PID==student1])
    student_l=[]
    for l in p1:
        for m in p2:
            student_l.append(relationship(l,m))

    max_score = np.max(np.array(student_l).astype(np.float))
    relationship_matrix.loc[student,student1] = max_score

print relationship_matrix

Answer 1

There are some "optimisations" you can do, here are a few examples I can quickly spot (assuming code1_ancestors , code2_ancestors , etc. are lists or something equivalent):

common_ancestors = []
for a1 in code1_ancestors:
    for a2 in code2_ancestors:
        if a1==a2:
            common_ancestors.append(a1)

can be made way faster by:

set(code1_ancestors)&set(code2_ancestors)

and to mention that your for loops actually can end up with duplicates of the common ancestors.

Or this:

code_descendants=[]
for gd in ca.get_descendants():
    code_descendants.append(gd.name)

can be improved by:

code_descendants = [gf.name for in ca.get_descendants()]

Or this also:

frequency= 0
for k in code_descendants:
        frequency= frequency + code_count.Count[k]

can be turned to:

frequency = code_count.loc[code_descendants, "Count"].sum()

Basically try to avoid doing things iteratively, ie for loops, and try to work with operations done the whole of the numpy arrays (underlying structure of pandas data frames).

Answer 2

I don't see a declaration of a Tree class, yet one is referenced in the first two lines of relationship().

You should be getting a "NameError: name 't' is not defined"