Can the field names in a MongoDB document be queried, perhaps using aggregation?

Question

In this article from the MongoDB blog, " Schema Design for Time Series Data in MongoDB " the author proposed storing multiple time series values in a single document as numbered children of a base timestamp (ie document per minute, seconds as array of values).

{
  timestamp_minute: ISODate("2013-10-10T23:06:00.000Z"),
  type: “memory_used”,
  values: {
    0: 999999,
    …  
    37: 1000000,
    38: 1500000,
    … 
    59: 2000000
  }
}

The proposed schema sounds like a good one but they fail to mention how to query the "values" field names which would be required if you wanted to know when the last sample occurred.

How would you go about constructing a query to find something like the time of the most recent metric (combining timestamp_minute and highest field name in the values)?

Thanks so much!

Answer 1

You can just query the minute document and then use a loop on the client to determine which timestamps have been set:

doc = c.find(...)
var last = 0
for (var i=0; i<60; i++)
    if (i in doc.values)
        last = i

Another approach which is a little more efficient is to use an array instead of a document for the per-second samples, and then use the length of the array to determine how many second samples have been stored:

doc = c.find(...)
last = doc.values.length - 1

Answer 2

I found the answer "can the field names be queried" in another blog post which showed iterating over the keys (as Bruce suggests) only doing so in a MapReduce function ala:

  var d = 0;
  for (var key in this.values)
      d = Math.max(d, parseInt(key));

---

For the MMS example schema (swapping in month for timestamp_minute and days in the values array labeled v below) here is the data and a query that produces the most recent metric date:

db.metricdata.find();

/* 0 */
{
    "_id" : ObjectId("5277e223be9974e8415f66f6"),
    "month" : ISODate("2013-10-01T04:00:00.000Z"),
    "type" : "ga-pv",
    "v" : {
        "10" : 57,
        "11" : 49,
        "12" : 91,
        "13" : 27,
      ...
    }
}

/* 1 */
{
    "_id" : ObjectId("5277e223be9974e8415f66f7"),
    "month" : ISODate("2013-11-01T04:00:00.000Z"),
    "type" : "ga-pv",
    "v" : {
        "1" : 145,
        "2" : 51,
        "3" : 63,
        "4" : 29
    }
}

And the map reduce function:

db.metricdata.mapReduce(
    function() {
        var y = this.month.getFullYear();
        var m = this.month.getMonth();
        var d = 0;

        // Here is where the field names used
        for (var key in this.v)
            d = Math.max(d, parseInt(key));

        emit(this._id, new Date(y,m,d));
    },
    function(key, val)
    {
        return null;
    },
    {out: "idandlastday"}
 ).find().sort({ value:-1}).limit(1)

This produces something like

/* 0 */
{
    "_id" : ObjectId("5277e223be9974e8415f66f7"),
    "value" : ISODate("2013-11-04T05:00:00.000Z")
}