Unable to convert UTF-8 characters - Python

Question

I receive a bunch of data into a variable using a mechanize and urllib in Python 2.7. However, certain characters are not decoded despite using .decode(UTF-8) . The full code is as follows:

#!/usr/bin/python

import urllib
import mechanize
from time import time

total_time = 0
count = 0
def send_this(url):
        global count
        count = count + 1
        this_browser=mechanize.Browser()
        this_browser.set_handle_robots(False)
        this_browser.addheaders=[('User-agent','Chrome')]

        translated=this_browser.open(url).read().decode("UTF-8")
        return translated

def collect_this(my_ltarget,my_lhome,data):
        global total_time
        data = data.replace(" ","%20")
        get_url="http://mymemory.translated.net/api/ajaxfetch?q="+data+"&langpair="+my_lhome+"|"+my_ltarget+"&mtonly=1"
        return send_this(get_url)

ctr = 0
print collect_this("hi-IN","en-GB","This is my first proper computer program.")

The output of the print statement is:

{"responseData":{"translatedText":"\u092f\u0939 \u092e\u0947\u0930\u093e \u092a\u0939
u0932\u093e \u0938\u092e\u0941\u091a\u093f\u0924 \u0915\u0902\u092a\u094d\u092f\u0942\u091f
\u0930 \u092a\u094d\u0930\u094b\u0917\u094d\u0930\u093e\u092e \u0939\u0948
\u0964"},"responseDetails":"","responseStatus":200,"matches":[{"id":0,"segment":"This is my 
first proper computer program.","translation":"\u092f\u0939 \u092e\u0947\u0930\u093e \u092a
\u0939\u0932\u093e \u0938\u092e\u0941\u091a\u093f\u0924 \u0915\u0902\u092a\u094d\u092f\u0942
\u091f\u0930 \u092a\u094d\u0930\u094b\u0917\u094d\u0930\u093e\u092e \u0939\u0948
\u0964","quality":"70","reference":"Machine Translation provided by Google, Microsoft, 
Worldlingo or MyMemory customized engine.","usage-count":0,"subject":"All","created-
by":"MT!","last-updated-by":"MT!","create-date":"2013-12-20","last-update-
date":"2013-12-20","match":0.85}]}

The characters starting with \\u...\u003c/code> are supposed to be the characters that were supposed to be converted.

Where have I gone wrong?

Answer 1

You don't have a UTF-8-encoded string. You have JSON with JSON unicode escapes in it. Decode it with a JSON decoder:

import json
json.loads(your_json_string)