How do I make raw unicode encoded content readable?

Question

I used net/http request a web API and the server returned a JSON response. When I print the response body, it displayed as raw ASCII content. I tried using bufio.ScanRunes to parse the content but failed.

I also tried write a simple server and return a unicode string and it worked well.

Here is the core code:

func (c ClientInfo) Request(method string, url string, form url.Values) string {
    req, _ := http.NewRequest(method, url, strings.NewReader(c.Encode(form)))
    req.Header = c.Header
    req.AddCookie(&c.Cookie)
    resp, err := http.DefaultClient.Do(req)
    defer resp.Body.Close()
    if err != nil {
        fmt.Println(err)
    }

    scanner := bufio.NewScanner(resp.Body)
    scanner.Split(bufio.ScanRunes)

    var buf bytes.Buffer
    for scanner.Scan() {
        buf.WriteString(scanner.Text())
    }
    rv := buf.String()
    fmt.Println(rv)
    return rv
}

Here is the example output:

{"forum":{"id":"3251718","name":"\合\肥\工\业\大\学\宣\城\校\区","first_class":"\高\等\院\校","second_class":"\安\徽\院\校","is_like":"0","user_level":"1","level_id":"1","level_name":"\素\未\谋\面","cur_score":"0","levelup_score":"5","member_num":"80329","is_exists":"1","thread_num":"108762","post_num":"3445881","good_classify":[{"class_id":"0","class_name":"\全\部"},{"class_id":"1","class_name":"\公\告\类"},{"class_id":"2","class_name":"\吧\友\专\区"},{"class_id":"4","class_name":"\活\动\专\区"},{"class_id":"6","class_name":"\社\团\班\级"},{"class_id":"5","class_name":"\资\源\共\享"},{"class_id":"8","class_name":"\温\馨\生\活\类"},{"class_id":"7","class_name":"\咨\询\新\闻\类"},{"class_id":"3","class_name":"\风\采\展\示\区"}],"managers":[{"id":"793092593","name":"yi\抹\明\媚\的\忧\伤"},

...

Answer 1

That is just the standard way to escape any Unicode character.

Unmarshal it to see the unquoted text (the json package will unquote it):

func main() {
    var i interface{}
    err := json.Unmarshal([]byte(src), &i)
    fmt.Println(err, i)
}

const src = `{"forum":{"id":"3251718","name":"\u5408\u80a5\u5de5\u4e1a\u5927\u5b66\u5ba3\u57ce\u6821\u533a","first_class":"\u9ad8\u7b49\u9662\u6821","second_class":"\u5b89\u5fbd\u9662\u6821","is_like":"0","user_level":"1","level_id":"1","level_name":"\u7d20\u672a\u8c0b\u9762","cur_score":"0","levelup_score":"5","member_num":"80329","is_exists":"1","thread_num":"108762","post_num":"3445881","good_classify":[{"class_id":"0","class_name":"\u5168\u90e8"},{"class_id":"1","class_name":"\u516c\u544a\u7c7b"},{"class_id":"2","class_name":"\u5427\u53cb\u4e13\u533a"},{"class_id":"4","class_name":"\u6d3b\u52a8\u4e13\u533a"},{"class_id":"6","class_name":"\u793e\u56e2\u73ed\u7ea7"},{"class_id":"5","class_name":"\u8d44\u6e90\u5171\u4eab"},{"class_id":"8","class_name":"\u6e29\u99a8\u751f\u6d3b\u7c7b"},{"class_id":"7","class_name":"\u54a8\u8be2\u65b0\u95fb\u7c7b"},{"class_id":"3","class_name":"\u98ce\u91c7\u5c55\u793a\u533a"}]}}`

Output (trimmed) (try it on the Go Playground ):

<nil> map[forum:map[levelup_score:5 is_exists:1 post_num:3445881 good_classify:[map[class_id:0 class_name:全部] map[class_id:1 class_name:公告类] map[class_id:2 class_name:吧友专区] map[class_id:4 class_name:活动专区] map[class_id:6 class_name:社团班级] map[class_id:5 class_name:资源共享] map[class_id:8 class_name:温馨生活类] map[class_name:咨询新闻类 class_id:7] map[class_id:3 class_name:风采展示区]] id:3251718 is_like:0 cur_score:0

If you just want to unquote a fragment, you may use strconv.Unquote() :

fmt.Println(strconv.Unquote(`"\u7d20\u672a\u8c0b"`))

Output (try it on the Go Playground ):

素未谋 <nil>

Note that strconv.Unquote() expects a string that is in quotes, that's why I used a raw string literal, so I could add quotes, and also so that the compiler itself will not interpret / unquote the Unicode escapes.

See related question: How to convert escape characters in HTML tags?