How do I read a streaming response body using Golang's net/http package?

Question

I am trying to connect to an endpoint that does http streaming of json data. I was wondering how to perform a basic request using Go's net/http package and read the response as it comes in. Currently, I am only able to read the response when the connection closes.

resp, err := http.Get("localhost:8080/stream")
if err != nil {
    ...
}
...
// perform work while connected and getting data

Any insight would be greatly appreciated!

Thanks!

-RC

Answer 1

The answer provided by Eve Freeman is the correct way to read json data. For reading any type of data, you can use the method below:

resp, err := http.Get("http://localhost:3000/stream")
...

reader := bufio.NewReader(resp.Body)
for {
    line, err := reader.ReadBytes('\n')
    ...

    log.Println(string(line))
}

Answer 2

The way to do streaming JSON parsing is with a Decoder:

json.NewDecoder(resp.Body).Decode(&yourStuff)

For a streaming API where it's a bunch of objects coming back (a la Twitter), that should stream great with this model and the built-in encoding/json API. But if it's a large response where you have an object that's got a giant array with 10MB of stuff, you probably need to write your own Decoder to pull those inner pieces out and return them. I'm running into that problem with a library I've written.