如何以更有效的方式比较Python中的两个文件？

Question

I have to compare two large files, but I'm having some problems related to performance.

So, let's consider two files X and Y .

X has 42000 records. One word per line.

Y has 881000. Three words per line, ie three columns.

I want to compare the words of the X file with the first word of the Y file.

If I find the X_word in the Y_first_column_word , then I write the the word of the second column of the Y file to a file(Y_second_column_word) .

See the code:

to_file = open( output_file, 'w' )                # opening the file to write
f1      = open( input_file1, "rU" ).readlines()   # reading 1st file  42000 records
f2      = open( input_file2, "rU" ).readlines()   # reading 2nd file 881000 records

for i, w1 in enumerate( f1 ):
    for j, line in enumerate( f2 ):
        w2 = line.split(',')                      # splitting words from  2nd file
        if w1.strip() == w2[0].strip():           # removing trails
            if w2[1].strip() == '':               # when it is blank, get 1st column word 
                w2[1] = w2[0]
            print>>to_file, w2[1]

to_file.close()                                   # closing the file

I've carried out tests run with test data, and it does what I want. But when I run it with the real data it becomes unresponsive. My last try spent 18 hours.

Is there any way to improve this code to get it running in a more efficient way?

Answer 1

Your current approach is O(N**2) , if you use a dictionary to store the content of second file then you can do this in linear time.

with open(input_file1, "rU")as f1, open(input_file2, "rU") as f2:
    words_dict = {k:v for k, v, _ in (line.split(',') for line in f2)}
    for word in f1:
        word = word.rstrip()
        if word in words_dict:
           #write words_dict[word] to to_file