如何处理FileInput中无效的utf8？

Question

I have basically the following code:

def main():
    for filename in fileinput.input():
        filename = filename.strip()
        process_file(filename)

The script takes a newline-separated list of file names as its input. However, some of the file names contain invalid utf8, which causes fileinput.input() to implode. I've read about the surrogateescape error handler, which I think is what I want, but I don't know how to set the error handler for fileinput.

In short: how do I get fileinput to deal with invalid Unicode?

Answer 1

filenames on POSIX may be arbitrary sequences of bytes (except b'\\0' and b'/' ) ie, no character encoding can decode them in the general case (that is why os.fsdecode() exists that uses surrogateescape error handler).

You could use a binary mode to read the filenames then either skip undecodable filenames if the input shouldn't contain them or pass them as is (or os.fsdecode() ) to functions that expect filenames:

for filename in fileinput.input(mode='rb'):
    process_file(os.fsdecode(filename).strip())

Beware, there were several known Python bugs related to using a binary mode and fileinput eg:

• fileinput should use stdin.buffer for "rb" mode
• fileinput.FileInput.readline() always returns str object at the end even if in 'rb' mode

Answer 2

Following documentation please use opening hook:

def main():
for filename in fileinput.input(openhook=fileinput.hook_encoded("utf-8")):
    filename = filename.strip()
    process_file(filename)