在删除或替换指定字符的同时读取文件？

Question

I have a large file that contains some NULL characters. I'd like to read this file in Python, as if these NULLs aren't there. I could read the entire file into an in-memory string and do a str.replace , but this is inefficient, especially given its total size (which can be in the multiple GBs).

Is there an efficient way to read a file in Python, while dynamically dropping certain characters, or replacing them with others?

Answer 1

Open the file in binary mode and read it in chunks of suitable size. Remove from each chunk undesired characters and write the resulting bytes to another file opened for writing.

This will work for \\x00 bytes, but will certainly fail if it's a text file with utf-8 encoding, where a single letter can take several bytes.

This can be solved usingcodecs.open . The returned file-like object allows you to read approximate number of bytes in the given encoding.