Aggregation the data from mongodb, map reduce or any other ways?

Question

Well i am struggling with the aggregation problems. I thought the easiest way to solve problem is to use map reduce or make separate find queries and then loop through with the async library help.

The schema is here:

db.keyword
keyword: String
start: Date
source: String(Only one of these (‘google’,’yahoo’,’bing’,’duckduckgo’) )
job: ref db.job
results: [
    {
            title: String
            url: String
            position: Number
    }
]


db.job
name: String
keywords: [ String ]
urls: [ String ]
sources: [ String(‘google’,’yahoo’,’bing’,’duckduckgo’) ]

Now i need to take the data to this form:

data = {
    categories: [ 'keyword1', 'keyword2', 'keyword3' ],
    series: [
        {
            name: 'google',
            data: [33, 43, 22]
        },
        {
            name: 'yahoo',
            data: [12, 5, 3]
        }

    ]
}

Well the biggest problem is that the series[0].data array is made of really difficult find, matching the db.job.urls against the db.keyword.results.url and then get the position. Is there any way to simplify the query_? I have looked through many of the map reduce examples, but I cant find the correct way what data to map and which to reduce.

Answer 1

It looks as though you are trying to combine data from two separate collections (keyword and job).

Map Reduce as well as the new Aggregation Framework can only operate on a single collection at a time.

Your best bet is probably to query each collection separately and programmatically combine the results, saving them in whichever form is best suited to your application.

If you would like to experiment with Map Reduce, here is a link to a blog post written by a user who used an incremental Map Reduce operation to combine values from two collections.
http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/

For more information on using Map Reduce with MongoDB, please see the Mongo Documentation: http://www.mongodb.org/display/DOCS/MapReduce (The section on incremental Map Reduce is here: http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-IncrementalMapreduce )

There are some additional Map Reduce examples in the MongoDB Cookbook: http://cookbook.mongodb.org/

For a step-by-step walkthrough of how a Map Reduce operation is run, please see the "Extras" section of the MongoDB Cookbook recipe "Finding Max And Min Values with Versioned Documents" http://cookbook.mongodb.org/patterns/finding_max_and_min/

Hopefully the above will give you some ideas for how to achieve your desired results. As I mentioned, I believe that the most straightforward solution is simply to combine the results programmatically. However, if you are successful writing a Map Reduce operation that does this, please post your solution, so that the Community may gain the benefit of your experience.