使用 base64 解码大文件时出现内存错误

Question

I am trying to decode string which is stored in a file. The file size is around 300 MB. It's throwing MemoryError while decoding.

base64.b64decode(bytes(file_content))

Is there any solution for this.

Answer 1

If your input data isn't already bytes , you've just forced an unnecessary copy to make a true bytes object. Annoyingly, it looks like base64.b64decode will perform this unnecessary conversion for you for all inputs aside from bytearray , even though the underlying API ( binascii.a2b_base64 ) properly supports the buffer protocol (so it works just fine on, say, mmapped files).

So if you want to avoid an unnecessary copy, change to:

binascii.a2b_base64(file_content)

which decodes without copying the input at all.

The other general tip is to make sure you're running a 64 bit build of Python (the default suggested installer for Windows is still 32 bit, sadly). When you're talking hundreds of MB of data, with copies of it floating around, it's pretty easy to hit the 2 GB limit on user mode virtual address space, and upgrading to 64 bit Python will fix that for you (your code might be slow if you don't have enough RAM for it, but it shouldn't die from a MemoryError so easily).