How do I iterate through a large file without running out of memory in python?

Question

What I want to do is basically print how many times a string appears in a file, but the file is so large that the program always crashes when I try to iterate though it:

import gzip
res = []
def fileopener(f):
    with gzip.open(f) as fhand:
        for line in fhand:
            res.append(line.count(b'NEU'))
        print(sum(res))

The expected results would be the total sum of 'NEU', but instead the program crashes before it can produce any output. Is there something I could do that would stop this from happening?

Answer 1

Keep a running total as you iterate instead of simply accumulating things to add. The built-in function sum will do this for you.

with gzip.open(f) as fhand:
    result = sum(line.count(b'NEU') for line in fhand)