Problem with Unicode in Python as a procmail filter

Question

I'm running into issues when trying to process mail with procmail and Python. I am using syntax something like this:

:0
...[Filter] | (python3 script.py) >> file.txt

as procmail syntax. My Python script extracts the mail from stdin, converts MIME to unicode and outputs it to a file as follows:

def main():
        dataset = Data()
        indata = (Parser().parse(sys.stdin)).as_string()

        indata = (quopri.decodestring(indata)).decode('utf-8')

        arrayofstrings = indata.split("\n")

        for line in arrayofstrings:
                [write some data to <dataset>]
        filename = "outfile.txt"
        file = open(filename, "w")
        file.write(dataset.toString())

Data() is a structure that stores a series of unicode strings and toString() concatenates them.

If I run this script in bash with a stored mail like this:

cat test.txt | python3 script.py

it correctly writes the data as unicode to the file.

However, if I get a mail and it gets processed, procmail writes the following error to the log:

UnicodeEncodeError: 'ascii' codec can't encode character '\\xdf' in position 83: ordinal not in range(128)

If I change the last line of the python script to:

file.write(dataset.toString().encode('utf-8'))

I get the correctly encoded string in the file. I want it in unicode though.

Answer 1

The immediate problem is that Python sets up its system encoding based on the terminal it is connected to; but of course, when you run it from Procmail, it is not connected to a terminal at all.

The workaround might involve setting PYTHONIOENCODING in your Procmail file:

:0
...[Filter] | PYTHONIOENCODING=utf-8 python3 script.py >> file.txt

(Note also that no parentheses are required; they would run Python in a subshell, but there seems to be no reason to want it to run in a subshell here.)

Notice, though, that this is not functionally different from specifically encoding the output into UTF-8 within your script (though I can see how you might want to avoid hardcoding it). Text in a file cannot be "Unicode" without also being serialized into an encoding. (And if you don't want UTF-8, you have to specify what you do want ... UTF-16le maybe? That would be compatible with Java and legacy Windows.)