How to handle UnicodeEncodeError in python script?

Question

I have a python script from which I am generating list of softwares installed on my machine. Let the name of this script be 'install.py' - it looks as below:

import wmi
w = wmi.WMI()
for p in w.Win32_Product():
    if (p.Version is not None) and (p.Caption is not None):
        print  p.Caption + " & "+ p.Version + "\\\\"
        print "\hline"

Now I am actually writting the output of this script into a output.tex file by executing it from some another script say "output_file.py", looks as below:

with open("D:/output.tex", "w+") as output:
    process = sp.call(["python", "D:/install.py"], stdout=output)

So when the above piece is executed I do get output in "output.tex" but along with error as :

UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 43:
ordinal not in range(128)

So, actually don't get details of all software on my system. So what shall I do to remove this error in my script. Kindly help.

Answer 1

The immediate issue is that Python 2 uses ascii encoding ( sys.getdefaultencoding() ) when sys.stdout is redirected. You could override it with PYTHONIOENCODING envvar:

call([sys.executable, os.path.join(script_dir, 'install.py')], stdout=file,
     env=dict(os.environ, PYTHONIOENCODING='utf-8'))

It would be enough on *nix system but Windows may interfere with passing bytes between install.py and the file (eg, the pipe | is broken for binary content in PowerShell ).

To workaround it, you could pass the filename as the command-line parameter to install.py and write to the file instead of printing to sys.stdout there.

The right solution is to put the necessary functionality into a function and import the module instead of running it as a subprocess . You can use multiprocessing if you want to run the code in a different process as shown in the link .