Python readline（）和Counter在很长的一行上导致MemoryError

Question

I'm having the issue of a memory error.

pifile = 'pibillion.txt'
with open(pifile, "r+") as a:
    data = str(a.readline())
    c = Counter(data)

All my code does is read one very very large line of the digits of pi. The txt file is only 953 MB. I have 8 GB RAM. I'm guessing the error is that it runs into the String size limitation but I'm not sure. The rest of the code inserts a line break at increments of two. Any help would be greatly appreciated as to how to continue with this.

The exact error I'm getting is this:

data = str(a.readline())
   MemoryError

Answer 1

Python is not inherently lazy (like haskell), so reading a string will put it all in memory. Add to that some string conversions and you're out of memory. Instead, do this iteratively, like the following.

Note that I have used a new file, as files are usually stored contiguously, so inserting is very expensive.

with open('pibillion.txt', 'r') as old_file, open('pibillion_.txt', 'w') as new_file:
    while True:
        c = old_file.read(2)
        if not c:
            break
        new_file.write(c + '\n')