在python中读取大数据文件的最快方法

Question

I have got some (about 60) huge (>2 gig) CSV files which I want to loop through to to make subselections (eg each file contains data of 1 month of various financial products, i want to make 60-month time series of each product) .

Reading an entire file into memory (eg by loading the file in excel or matlab) is unworkable, so my initial search on stackoverflow made me try python. My strategy was to loop through each line iteratively and write it away in some folder. This strategy works fine, but it is extremely slow.

From my understanding there is a trade-off between memory usage and computation speed. Where loading the entire file in memory is one end of the spectrum (computer crashes), loading a single line unto the memory each time is obviously on the other end (computation time is about 5 hours).

So my main question is: *Is there a way that to load multiple lines into memory, as to do this process (100 times?) faster. While not losing functionality? * And if so, how would I implement this? Or am I going about this all wrong? Mind you, below is just a simplified code of what I am trying to do (I might want to make subselections in other dimensions than time). Assume that the original data files have no meaningful ordering (other than they being split into 60 files for each month).

The method in particular I am trying is:

#Creates a time series per bond
import csv
import linecache


#I have a row of comma-seperated bond-identifiers 'allBonds.txt' for each month
#I have 60 large files financialData_&month&year


filedoc=[];
months=['jan','feb','mar','apr','may','jun','jul','aug','sep','oct','nov','dec'];
years=['08','09','10','11','12'];
bonds=[];


for j in range(0,5):
     for i in range(0,12):    
         filedoc.append('financialData_' +str(months[i]) + str(years[j])+ '.txt')




for x in range (0,60):
line = linecache.getline('allBonds.txt', x)  
bonds=line.split(','); #generate the identifiers for this particular month
with open(filedoc[x]) as text_file:

     for line in text_file:

          temp=line.split(';');

          if temp[2] in bonds: : #checks if the bond of this iteration is among those we search for
               output_file =open('monthOutput'+str(temp[2])+ str(filedoc[x]) +'.txt', 'a')
               datawriter = csv.writer(output_file,dialect='excel',delimiter='^', quoting=csv.QUOTE_MINIMAL)
               datawriter.writerow(temp)
               output_file.close()

Thanks in advance.

Ps Just to make sure: the code works at the moment (though any suggestions are welcome of course), but the issue is speed.

Answer 1

I would test pandas.read_csv mentioned in https://softwarerecs.stackexchange.com/questions/7463/fastest-python-library-to-read-a-csv-file . It supports reading the file in chunks ( iterator=True option)

I think this part of your code may cause serious performance problems if the condition is matched frequently.

if temp[2] in bonds: : #checks if the bond of this iteration is among those we search for
    output_file = open('monthOutput'+str(temp[2])+ str(filedoc[x]) +'.txt', 'a')
    datawriter = csv.writer(output_file,dialect='excel',delimiter='^', 
                            quoting=csv.QUOTE_MINIMAL)
    datawriter.writerow(temp)
    output_file.close()

It would be better to avoid opening a file, creating a cvs.writer() object and then closing the file inside a loop.