即使所有内容都是unicode（python 2.7），ascii也会解码错误

Question

i am running a script in dataflow (apache beam) it runs in python 2.7.12 and does some text processing with unicode strings.

Amongst the processing i do the following, where noun and phrase are unicode ( i think... )

# -*- coding: utf-8 -*-
...
key = u"{}_{}".format(
    noun, phrase.replace(u" ", u"_")
)

However it yields ascii decode errors

'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)

I can put in debugging and get a repr of the strings used in as noun and phrase but i currently don't have them since my logging didn't output them.

i don't understand the ascii decode error when i think i am pretty specific that i want everything in unicode!

can you give some hints or should i come back with more info about the input strings?

Answer 1

OK, so you have a non ascii character in your string. You need to convert phrase into unicode directly

 phrase.decode('latin-1')

before manipulating in unicode.format

Answer 2

a colleague reminded me that i could always just decode the whole output, in this case being the key, to whatever format i chose.

key = u"{}_{}_{}_{}".format(
     business_unit_id, date, noun, phrase.replace(u" ", u"_")
    ).encode('ascii', 'ignore')

in the case i wanted ascii output and not care about missing chars like 💩.

i could also use ...).encode('utf-8') if i wanted that output in unicode.

in my case i settled with ascii output as the pipeline in apache beam did not seem happy with unicode keys in its map reduce pipelines