我使用 pprof 进行的 golang 程序分析显示，std/json 包中 json (* decodeState) objectInterface 的内存增加

Question

I have a golang program that uses unmarshall from std "encoding/json" package keeps increasing in size (memory leak). A diagram of memory profile using pprof shows memory increasing at json (* decodeState) objectInterface. I want to understand how and why it could be happening to fix the issue.

I have tried several things at upper level like release of returned value to avoid the leak, but with no success.

func (j JSONEncoding) From(b []byte, msg interface{}) (interface{}, error) {
    err := json.Unmarshal(b, &msg)
    return msg, err
}

pprof top5 shows this call, and the details below;

4096.89kB 24.43% 24.43%  4096.89kB 24.43%  encoding/json.(*decodeState).objectInterface

(pprof) list objectInterface
Total: 20.65MB
ROUTINE ======================== encoding/json.(*decodeState).objectInterface in /usr/local/go/src/encoding/json/decode.go
       6MB     7.50MB (flat, cum) 36.32% of Total
         .          .   1061:   return v
         .          .   1062:}
         .          .   1063:
         .          .   1064:// objectInterface is like object but returns map[string]interface{}.
         .          .   1065:func (d *decodeState) objectInterface() map[string]interface{} {
       3MB        3MB   1066:   m := make(map[string]interface{})
         .          .   1067:   for {
         .          .   1068:       // Read opening " of string key or closing }.
.
. deleted code
.
         .          .   1095:
         .          .   1096:       // Read value.
       3MB     4.50MB   1097:       m[key] = d.valueInterface()

using topN, and using visualization I can see this box increasing with time/processing.

This unmarshall is called in a loop but nothing is saved that could be a reason for the leak. I am not sure how and what of something to be done so that this leak is avoided.

Update : The memory leak was more of a memory accrual, in the code at some other place. While trying to write a minimum code that shows the problem, I was not able to reproduce, and had to dig all the code to find out that an internal library was using a map to cache and the cache cleaner was not working properly.

My problem was that pprof is giving info on who is allocating, but not where it is being kept. I think my question should have been, how can we find out which objects have references of what size. I know that one allocated reference could be in multiple places but such information would easily help in finding this kind of issue.

Answer 1

Q: Why don't you do something like this:

https://golang.org/pkg/encoding/json/

func (a *Animal) UnmarshalJSON(b []byte) error {
    var s string
    if err := json.Unmarshal(b, &s); err != nil {
        return err
    }
    switch strings.ToLower(s) {
    default:
        *a = Unknown
    case "gopher":
        *a = Gopher
    case "zebra":
        *a = Zebra
    }

    return nil
}

In other words, is your implementation preventing objects (like "err", for example) from being garbage collected?