Best way to store golang map[string]interface into data store

Question

Scenario : I am having a arbitrary JSON size ranging from 300 KB - 6.5 MB read from MongoDB. Since it is very much arbitrary/dynamic data I cannot have struct type defined in golang, So I am using map[sting]interface{} type. And string of JSON data is parsed by using encoding/json 's Unmarshal method. Some what similar to what is mentioned in Generic JSON with interface{} .

Issue : But the problem is its taking more time (around 30ms to 180ms) to parse the string json into map[string]interface{} . (Comparing to php parsing json using json_encode/decode / igbinary/ msgpack)

Question : Is there any way to pre-process it and store in cache?

I mean parse string into map[string]interface{} and serialize it and store to some cache, then when we retrieve it should not take much time to unserialization and proceed with execution.

Note: I am Newbie for the golang any suggestion are highly appriciated. Thanks

Updates : Serialization using Gob , binary built-in package & Msgpack implementation for Golang package are already tried. No luck, No improvement in the time to unserialization.

Answer 1

The standard library package for JSON is notoriously slow. There is a good reason for that: it use RTTI to provide a really flexible interface that is really simple. Hence the unmarshalling being slower than PHP's…

Fortunately, there is an alternative way, which is to implement the json.Unmarshaller interface on the types you want to use. If this interface is implemented, the package will use it instead of its standard method so you can see huge performance boosts.

And to help you, there is a small group of tools that appeared, among which:

• https://godoc.org/github.com/benbjohnson/megajson
• https://godoc.org/github.com/pquerna/ffjson

(listing here the main players from memory, there must be others)

These tools will generate tailored implementations of the json.Unmarshaller interface for the types you requested. And with go:generate , you can even integrate them seamlessly to your build step.