python and ipython seem to treat unicode characters differently

Question

I'm running python 2.7.12 from anaconda on windows 10. Included in the distro is ipython 5.1.0. I wrote a program to print certain columns of queried rows in a mysql database. The columns contain strings in unicode. When the program is run in python, an exception is thrown when a unicode character in one of the strings is first seen. The same program in ipython works, displaying all characters appropriately.

I've distilled the issue into a separate little program as follows:

name = u'O\u2019Connor'
try:
   print name
except:
   print "exception 1 thrown"

try:
   print u"{}".format(name)
except:
   print "exception 2 thrown"

try:
   print u"%s" % name
except:
   print 'exception 3 thrown'

When run using python, exceptions are thrown everytime. When run in ipython, all three print statements work. Obviously, there is a difference between the two versions in the way unicode is handled. What is the difference and what should I do so that my program will handle being run in either environment?

Answer 1

Looks like ipython is using a sane default output encoding (probably UTF-8 or UTF-16), while plain Python is using cp437 , a limited one-byte-per-character ASCII superset that can't represent the whole Unicode range.

If you can control the command prompt, you can run chcp 65001 before launching Python to make it use the "code page" for UTF-8 (which Python should pick up on). You might want to make this the default for command prompts in general to avoid future problems .