JSON on Golang - Unmarshal Graphite Data

Question

I am playing around with Golang and JSON trying to do some calculations with data extracted from Graphite API.

For simplicity, a snippet of the data sent by Graphite is:

[
{
    "target": "server1.loadavg.1min",
    "datapoints": [
        [
            0.16,
            1422770850
        ],
        [
            0.16,
            1422770880
        ],
        [
            null,
            1422771120
        ]
    ]
},
{
    "target": "server2.loadavg.1min",
    "datapoints": [
        [
            0.19,
            1422770850
        ],
        [
            null,
            1422771390
        ],
        [
            0.14,
            1422771420
        ]
    ]
}
]

I've been reading through the go json tutorial about how to use a generic interface{} for JSON arbitrary data, but I'm struggling with some aspects of the process.

I've tried to define a structure that will hold this data, read the file contents and unmarshal it to this structure:

type Graphite struct {
  Metric struct {
    Target     string      `json:"target"`
    Datapoints [][]float64 `json:"datapoints"`
  }
}

var results []Graphite
err = json.Unmarshal(d, &r)
if err != nil {
    panic(err)
}
fmt.Printf("%v\n", r)

But the result is:

[{{ []}} {{ []}}]

I could do it of course with a generic interface{} but I would like to know what am I missing here.

Could you please help me?

Thank you!

Answer 1

I like to start at the simplest type and work my way out. First you need to represent your data point.

type DataPoint []float64

Then a metric is just a target and a series of data points.

type Metric struct {
    Target string      `json:"target"`
    Points []DataPoint `json:"datapoints"`
}

There is no need for your Graphite struct. Your JSON is just a JSON array of Metric s.

var results []Metric
err := json.Unmarshal([]byte(data), &results)

Here's a playground link with a complete example.

Answer 2

The problem with the above answer is it turns null points into points with value 0, which is not correct. Null means "not known". Some people suggest using pointers to floats so nil pointer means "no value", but that has a significant overhead (eg 8bytes on most 64bit platforms, not to mention the memory dereferencing overhead).

Better to use golang's math NaN support to mark nulls, which requires no additional data as it's built into the float representation. You can do this with a custom Unmarshal function like so:

type Metric struct {
    Target     string
    Datapoints []Point
}

type Point struct {
    Val float64
    Ts  uint32
}

var errInvalidFormat = errors.New("invalid format")

func (p *Point) UnmarshalJSON(data []byte) error {
    if len(data) < 2 {
        return errInvalidFormat
    }
    // find first digit or 'n' for "null"
    for (data[0] < 48 || data[0] > 57) && data[0] != 110 {
        if len(data) == 1 {
            return errInvalidFormat
        }
        data = data[1:]
    }
    // find comma
    var i int
    for i = 0; i < len(data); i++ {
        if data[i] == 44 {
            break
        }
    }
    if i == 0 {
        return errInvalidFormat
    }

    if bytes.HasPrefix(data[:i], []byte("null")) {
        p.Val = math.NaN()
    } else {
        fl, err := strconv.ParseFloat(string(data[:i]), 64)
        if err != nil {
            return err
        }
        p.Val = fl
    }
    data = data[i:]
    if len(data) < 2 {
        return errInvalidFormat
    }

    // find first digit
    for (data[0] < 48 || data[0] > 57) && data[0] != 110 {
        if len(data) == 1 {
            return errInvalidFormat
        }
        data = data[1:]
    }
    // find last digit
    for i = 0; data[i] >= 48 && data[i] <= 57 && i < len(data); i++ {
    }
    if i == 0 {
        return errInvalidFormat
    }

    ts, err := strconv.ParseUint(string(data[:i]), 10, 32)
    if err != nil {
        return err
    }
    p.Ts = uint32(ts)
    return nil
}

Full example program: on playground