如何在python中处理非常大的文件？

Question

This question was asked earlier, but quite a while ago. I am currently trying to open a very large file (20GB) to manipulate stuff.

I am using:

read_path = '../text/'
time = 3600
data = open(read_path+'genomes'+str(time)).read().replace(',','\n').replace('\n','')

and it works fine when I choose a smaller file in the same directory (genomes1000), but when I change the time to the one matching the larger file I get the error.

The exact error message is:

Tempo:analytics scottjg$ python genomeplot.py 
Traceback (most recent call last):
  File "genomeplot.py", line 27, in <module>
    data = open(read_path+'genomes'+str(time)).read().replace(',','\n').replace('\n','')
OSError: [Errno 22] Invalid argument
Thoughts?

Answer 1

Your code reads the total contents of the file into memory:

open(read_path+'genomes'+str(time)).read()

I suspect that you do not have memory available to accomodate this and that is probably the reason for the failure. Wouldn't it be better to process it line by line with a call to readline in a loop instead?