如何将数据读入Python但不能全部读入内存？

Question

I need to parse through a file of about 100,000 records. Is there a way to do this without loading the whole file into memory? Does the csv module already do this (ie, not load the entire file into memory)? If it matters, I plan on doing this in IDLE.

Answer 1

I've never used the cvs module, but you'll want to look into using a generator, this will allow you to process a record at a time without reading the entire file in at once. For example, with a file, you can do something like...

def read_file(some_file):
    for line in open(some_file):
        yield line

all_lines = read_file("foo")
results = process(all_lines)

The all_lines will be a generator and will return one line each time it is referenced, as in:

for line in all_lines:
    ...

I'd imagine you can do this with the cvs module as well.