Go unmarshalling JSON from compessed HTTP: invalid character looking for beginning of value

Question

I've just written my first Go application which downloads and unmarshales simple JSON object over http. Http content is compressed: 'content-encoding': 'deflate'

I used several well-known examples (like this ). Unfortunately the application fails to parse the desired JSON with quite rare and strange error. I wasn't able to find out what's the problem. Any help will be appreciated.

JSON input (Python was used for debugging)

In [8]: r = requests.get("http://172.17.0.31:20000/top")

In [9]: r.text
Out[9]: u'{"timestamp":{"tv_sec":1428447555,"tv_usec":600186},"string_timestamp":"2015-04-07 22:59:15.600186","monitor_status":"enabled"}'
In [18]: r.headers
Out[18]: {'content-length': '111', 'content-type': 'application/json', 'connection': 'close', 'content-encoding': 'deflate'}

Source code ( UPDATED according to the answers)

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string    `json:"string_timestamp"`
    Monitor_status   string    `json:"monitor_status"`
}

type Timestamp struct {
    Tv_sec  int `json:"tv_sec"`
    Tv_usec int `json:"tv_usec"`
}

func get_content() {

    url := "http://172.17.0.31:20000/top"

    res, err := http.Get(url)
    if err != nil {
        panic(err.Error())
    }
    fmt.Println(res)

    body, err := ioutil.ReadAll(res.Body)
    if err != nil {
        panic(err.Error())
    }
    fmt.Println(body)

    var jsondata Top
    err = json.Unmarshal(body, &jsondata)
    if err != nil {
        panic(err.Error())
    }

    fmt.Println(jsondata)
}

func main() {
    get_content()
}

Error

[vitaly@thermaltake elliptics-manager]$ go run main.go 
&{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json] Content-Length:[111] Content-Encoding:[deflate]] 0xc20803e340 111 [] true map[] 0xc208028820 <nil>}
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
panic: invalid character 'x' looking for beginning of value

UPD: Thanks everyone. Now it's obvious that the reason of this issue was a deflate compression of HTTP response. However, it's still not clear how to perform a decompression in Golang (see here ).

Answer 1

The Go JSON marshaller can only marshal unicode strings. It seems that your JSON is not encoded in unicode, but with some other encoding (deflate?).

If you take your bytes stream:

[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]

And try to get a unicode string out of it:

body := []byte{120, 156, 77, 203, 65, 14, 130, 48, 16, 70, 225, 171, 152, 127, 93, 76, 59, 51, 162, 244, 50, 13, 96, 99, 154, 216, 98, 232, 192, 134, 112, 119, 81, 55, 110, 95, 190, 183, 65, 83, 142, 85, 251, 252, 130, 223, 160, 107, 168, 113, 132, 119, 66, 55, 145, 182, 117, 108, 62, 109, 249, 70, 98, 234, 108, 183, 27, 84, 157, 83, 121, 132, 191, 19, 100, 221, 165, 177, 210, 216, 235, 137, 200, 11, 123, 230, 243, 207, 195, 32, 79, 37, 233, 52, 135, 3, 235, 82, 15, 29, 75, 63, 60, 227, 29, 251, 27, 195, 90, 38, 189}
fmt.Println(string(body))

You would see a weird (compressed?) string in the console, not JSON.

I guess that the python http client automatically decompresses the deflated bytes while the Go http client does not (I know it does so for gzip but not sure if for deflate). You would have to read out the deflated bytes and convert them into a unicode string before you will be able to use the JSON marshaller to parse them.

Answer 2

I don't know about 'x', but struct fields must be public (start with a capital letter) to be considered by the json Unmarshaller. Of course then the names don't match the json keys, and you have to add json annotations like so:

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string `json:"string_timestamp"`
    Monitor_status   string `json:"monitor_status"`
}

Answer 3

I believe this is due to you double encoding. ioutil.ReadAll(res.Body) returns a []byte so when you do []byte(body) you're casting what is already a byte array, my guess is that first bytes UTF value is x . Just update this; json.Unmarshal([]byte(body), &jsondata) to json.Unmarshal(body, &jsondata) and I bet it will unmarshal just fine.

Also, unrelated to your error but as pointed out in the other answer if you do not export the fields in your struct (in go this means start the field name with a capital letter) then the unmarshaler will not be able to make user of them. To make that work you need to update you type to;

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string `json:"string_timestamp"`
    Monitor_status   string `json:"monitor_status"`
}

The json annotations are required because the unmarshaler is very strict and requires exact matches (case sensitive) for field names.

Answer 4

Try this please

func get_content() {

    url := "http://172.17.0.31:20000/top"

    res, err := http.Get(url)
    if err != nil {
        panic(err.Error())
    }
    defer res.Body.Close()

    fmt.Println("res body:", res.Body)

    body, err := ioutil.ReadAll(resp=.Body)

    fmt.Println("body:", body)
    re, err := zlib.NewReader(bytes.NewReader(body))
    fmt.Println("zlib:", re)
    enflated, err := ioutil.ReadAll(re)
    fmt.Println("enflated:", string(enflated))

    var jsondata Top
    err = json.Unmarshal(body, &jsondata)
    if err != nil {
        panic(err.Error())
    }

    fmt.Println(jsondata)
}

and ensure http://172.17.0.31:20000/top return json type.