'utf8' codec can't decode byte 0xdf in position 59: invalid continuation byte

Question

I have the following string which I am trying to embed in json:

mystr = '<url host="bing.com" method="GET" uri="/update?v=1&amp;uid=\xdf\xe2\x80|\xff\xff\xff\xff\xf3\x99\x83|\x88\xe2\x80|\xff\xff\xff\xff\xf6\x80|n\x80|&amp;os=45" user_agent=""/>\n    <url host="zaloopa.co.cc" method="GET" uri="/update?v=1&amp;uid=\xdf\xe2\x80|\xff\xff\xff\xff\xf3\x99\x83|\x88\xe2\x80|\xff\xff\xff\xff\xf6\x80|n\x80|&amp;os=45" user_agent=""/>'

import json
json.dumps('url':mystr)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

  File "/usr/lib64/python2.6/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.6/json/encoder.py", line 367, in encode
    chunks = list(self.iterencode(o))
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 294, in _iterencode
    yield encoder(o)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xdf in position 59: invalid continuation byte

I tried mystr.encode('ascii','ignore) but technically each of these values such as \\xdf are individual bytes themselves that are within the ascii range. Is there a way I can massage the data and feed into json without it crashing.

There is not hint of what the encoding is of that line is.

Thanks...Amro

Answer 1

mystr = u'<url host="bing.com" method="GET" uri="/update?v=1&amp;uid=\xdf\xe2\x80|\xff\xff\xff\xff\xf3\x99\x83|\x88\xe2\x80|\xff\xff\xff\xff\xf6\x80|n\x80|&amp;os=45" user_agent=""/>\n    <url host="zaloopa.co.cc" method="GET" uri="/update?v=1&amp;uid=\xdf\xe2\x80|\xff\xff\xff\xff\xf3\x99\x83|\x88\xe2\x80|\xff\xff\xff\xff\xf6\x80|n\x80|&amp;os=45" user_agent=""/>'

import json
print json.dumps({'url': mystr}, ensure_ascii=False)