Python unicode popen or Popen error reading unicode

Question

I have a program that generates the following output:

             ┌───────────────────────┐
             │10 day weather forecast│
             └───────────────────────┘
▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Tonight Sep 27      Clear               54      0 %
Tue Sep 28          Sunny               85/61   0 %
Wed Sep 29          Sunny               86/62   0 %
Thu Sep 30          Sunny               87/65   0 %
Fri Oct 01          Sunny               85/62   0 %
Sat Oct 02          Sunny               81/59   0 %
Sun Oct 03          Sunny               79/56   0 %
Mon Oct 04          Sunny               78/58   0 %
Tue Oct 05          Sunny               81/61   0 %
Wed Oct 06          Sunny               81/61   0 %

Last Updated Sep 27 10:20 p.m. CT
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔

This doesn't seem to format right on this site, but the lower lines at the top and the upper lines at the bottom result in a unicode error.

Here is the code example for os.popen

>>> buffer = popen('10day', 'r').read()
Traceback (most recent call last):
  File "/home/woodnt/python/10_day_forecast.py", line 129, in <module>
    line_lower(51)
  File "/home/woodnt/python/lib/box.py", line 24, in line_lower
    print upper_line * len
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-50: ordinal not in range(128)
>>> print buffer

             ┌───────────────────────┐
             │10 day weather forecast│
             └───────────────────────┘

>>> 

Here is the same for subprocess.Popen:

f = Popen('10day', stdout=PIPE, stdin=PIPE, stderr=PIPE)
o, er = f.communicate()
print o

             ┌───────────────────────┐
             │10 day weather forecast│
             └───────────────────────┘

print er
Traceback (most recent call last):
  File "/home/woodnt/python/10_day_forecast.py", line 129, in <module>
    line_lower(51)
  File "/home/woodnt/python/lib/box.py", line 24, in line_lower
    print upper_line * len
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-50: ordinal not in range(128)

Any ideas if this can be made to work without a lot of "under the hood" work? I'm just learning programming and starting with python

Answer 1

I'd say running your program from the console should work correctly because Python can guess the console encoding of the terminal window (cp437 on US Windows), but when run through a pipe Python uses the default of ascii. Try changing your program to encode all Unicode output to an explicit encoding, such as:

print (upper_line * len).encode('cp437')

Then when you read it from the pipe, you can either decode back to Unicode or print it directly to the terminal.