在mongodb上汇总大量数据

Question

Im looking for the most efficient way of performing summing queries against mongodb.

Currently we insert documents that contain various information and a date time stamp of when the document was created.

We need to sum this data to be viewed in the following ways:

Documents by hour of the day 1-24 Documents by day of the month 1-28/31 Documents by month of the year 1-12 Documents by year

This summed data will be accessed often as we're afraid that the massive amount of data thrown at mongo will have problems summing this data often.

We thought perhaps when a document is inserted into mongo that we have another document that contains these counts that we increment at the time of insertion. This way, we can quickly pull the counts without summing the data each request. Our concern is that this may not be the most efficient way to perform this type of operation in mongo

Any thoughts on the best way to accomplish this? My dev team as well as myself are new to mongodb and we want to make sure we don't fall into a performance trap with summing large sets of data.

Answer 1

The Aggregation Framework is perfectly suited for this type of queries.
I've done some examples for you below.

To start, let's populate some documents:

db.myDocumentCollection.insert({"date" : new Date('01/01/2012'), "topic" : "My Title 1"}); db.myDocumentCollection.insert({"date" : new Date('01/02/2012'), "topic" : "My Title 2"}); db.myDocumentCollection.insert({"date" : new Date('01/02/2012'), "topic" : "My Title 3"}); db.myDocumentCollection.insert({"date" : new Date('01/02/2012'), "topic" : "My Title 4"}); db.myDocumentCollection.insert({"date" : new Date('01/04/2012'), "topic" : "My Title 5"}); db.myDocumentCollection.insert({"date" : new Date('01/05/2012'), "topic" : "My Title 6"}); db.myDocumentCollection.insert({"date" : new Date('01/07/2013'), "topic" : "My Title 7"}); db.myDocumentCollection.insert({"date" : new Date('01/07/2013'), "topic" : "My Title 8"}); db.myDocumentCollection.insert({"date" : new Date('02/07/2013'), "topic" : "My Title 9"}); db.myDocumentCollection.insert({"date" : new Date('02/08/2013'), "topic" : "My Title 10"});

Return number of documents grouped by full date

db.myDocumentCollection.group(
{
   $keyf : function(doc) {
       return { "date" : doc.date.getDate()+"/"+doc.date.getMonth()+"/"+doc.date.getFullYear() };
    },
    initial: {count:0},
    reduce: function(obj, prev) { prev.count++; }
 })

Output

[
        {
                "date" : "1/0/2012",
                "count" : 1
        },
        {
                "date" : "2/0/2012",
                "count" : 3
        },
        {
                "date" : "4/0/2012",
                "count" : 1
        },
        {
                "date" : "5/0/2012",
                "count" : 1
        },
        {
                "date" : "7/0/2013",
                "count" : 2
        },
        {
                "date" : "7/1/2013",
                "count" : 1
        },
        {
                "date" : "8/1/2013",
                "count" : 1
        }
]

Return number of documents grouped by day of month for the year 2013

This is perhaps a little more relevant for the kinds of queries you want to do.
Here, we use the cond to specify only to group documents after 1/1/2013
You could use $gte and $lte to do date ranges here.

db.myDocumentCollection.group(
{
   $keyf : function(doc) {
       return { "date" : doc.date.getDate()+"/"+doc.date.getMonth()};
    },
    cond: {"date" : {"$gte": new Date('01/01/2013')}},
    initial: {count:0},
    reduce: function(obj, prev) { prev.count++; }
 })

Output

[
        {
                "date" : "7/0",
                "count" : 2
        },
        {
                "date" : "7/1",
                "count" : 1
        },
        {
                "date" : "8/1",
                "count" : 1
        }
]