Python urlencode特殊字符

Question

I have this variable here

reload(sys)
sys.setdefaultencoding('utf8') 
foo = u'"Esp\xc3\xadrito"'

which translates to "Espírito". But when I pass my variable to urlencode like this

urllib.urlencode({"q": foo}) # q=%22Esp%C3%83%C2%ADrito%22'

The special character is being "represented" wrongly in the URL.

How should I fix this?

Answer 1

You got the wrong encoding of "Espírito" , I don't know where you get that, but this is the right one:

>>> s = u'"Espírito"'
>>> 
>>> s
u'"Esp\xedrito"'

Then encoding your query:

>>> u.urlencode({'q':s.encode('utf-8')})
'q=%22Esp%C3%ADrito%22'

This should give you back the right encoding of your string.

EDIT: This is regarding right encoding of your query string, demo:

>>> s = u'"Espírito"'
>>> print s
"Espírito"
>>> s.encode('utf-8')
'"Esp\xc3\xadrito"'
>>> s.encode('latin-1')
'"Esp\xedrito"'
>>> 
>>> print "Esp\xc3\xadrito"
Espí­rito
>>> print "Esp\xedrito"
Espírito

This clearly shows that the right encoding for your string is most probably latin-1 (even cp1252 works as well), now as far as I understand, urlparse.parse_qs either assumes default encoding utf-8 or your system default encoding, which as per your post, you set it to utf-8 as well.

Interestingly, I was playing with the query you provided in your comment, I got this:

>>> q = "q=Esp%C3%ADrito"
>>> 
>>> p = urlparse.parse_qs(q)
>>> p['q'][0].decode('utf-8')
u'Esp\xedrito'
>>>
>>> p['q'][0].decode('latin-1')
u'Esp\xc3\xadrito'

#Clearly not ASCII encoding.
>>> p['q'][0].decode()

Traceback (most recent call last):
  File "<pyshell#320>", line 1, in <module>
    p['q'][0].decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
>>> 
>>> p['q'][0]
'Esp\xc3\xadrito'
>>> print p['q'][0]
Espírito
>>> print p['q'][0].decode('utf-8')
Espírito

Answer 2

urllib and urlparse appear to work with byte string in Python 2. To get unicode strings, encode and decode using utf-8.

Here's an example of a round-trip:

data = { 'q': u'Espírito'}

# to query string:
bdata = {k: v.encode('utf-8') for k, v in data.iteritems()}
qs = urllib.urlencode(bdata)

# qs = 'q=Esp%C3%ADrito'

# to dict:
bdata = urlparse.parse_qs(qs)
data = { k: map(lambda s: s.decode('utf-8'), v)
            for k, v in bdata.iteritems() }

# data = {'q': [u'Espídrito']}

Note the different meaning of escape sequences: in 'Esp\\xc3\\xadrito' (a string), they represent bytes, while in u'"Esp\\xedrito"' (a unicode object) they represent Unicode code points.