Popen.communicate() throws UnicodeDecodeError

Question

I have this code:

def __executeCommand(self, command: str, input: str = None) -> str:
    p = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE, universal_newlines=True)
    p.stdin.write(input)
    output, error = p.communicate()
    if (len(errors) > 0):
        raise EnvironmentError("Could not generate the key: " + error)
    elif (p.returncode != 0):
        raise EnvironmentError("Could not generate the key. Return Value: " + p.returncode)
    return output

Answer 1

univeral_newlines=True enables text mode. The subprocess output (bytes) is decoded using locale.getpreferredencoding(False) character encoding as @cdosborn mentioned .

If it doesn't work, provide the actual encoding that is used by command . And/or specify the error handler such as 'ignore' , 'surrogateescape' , etc as errors parameter:

from subprocess import Popen, PIPE

def __executeCommand(self, command: str, input: str = None, 
                     encoding=None, errors='strict') -> str:
    text_mode = (encoding is None)
    with Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE,
               universal_newlines=text_mode) as p:
        if input is not None and not text_mode:
            input = input.encode(encoding, errors) # convert to bytes
        output, err = p.communicate(input)
    if err or p.returncode != 0: 
        raise EnvironmentError("Could not generate the key. "
                               "Error: {}, Return Value: {}".format(
                                   ascii(err), p.returncode))
    return output if text_mode else output.decode(encoding, errors)

Answer 2

The universal_newlines=true setting results in an additional encoding that is the source of your error.

def __executeCommand(self, command: str, input: str = None) -> str:
    p = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE)
    output, error = p.communicate(input)
    if (len(errors) > 0):
        raise EnvironmentError("Could not generate the key: " + error)
    elif (p.returncode != 0):
        raise EnvironmentError("Could not generate the key. Return Value: " + p.returncode)
    return output

universal_newlines=true results in an encoding based on the output of:

python -c 'import locale; print locale.getpreferredencoding()'

Python threw an error when it expected your input to match the encoding above, but instead processed a byte clearly in a different encoding.

More info about python 3.4 universal_newlines here .

Answer 3

If you're using Python 3.6 or later, you can fix the error by changing this line:

p = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE, universal_newlines=True)

to this:

p = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE, encoding="utf-8", universal_newlines=True)

I used UTF-8 above, but you can substitute it with whatever encoding you need.

Answer 4

p = sub.Popen(cmd, stdout=sub.PIPE, stdin=sub.PIPE, stderr=sub.PIPE,
              text=True, encoding='utf-8')