MongoDB query on populated fields

Question

I have models called "Activities" that I am querying for (using Mongoose). Their schema looks like this:

var activitySchema = new mongoose.Schema({
    actor: {
        type: mongoose.Schema.ObjectId,
        ref: 'User',
        required: true
    },
    recipient: {
        type: mongoose.Schema.ObjectId,
        ref: 'User'
    },
    timestamp: {
        type: Date, 
        default: Date.now
    },
    activity: {
        type: String,
        required: true
    },
    event: {
        type: mongoose.Schema.ObjectId,
        ref: 'Event'
    },
    comment: {
        type: mongoose.Schema.ObjectId,
        ref: 'Comment'
    }
});

When I query for them, I am populating the actor , recipient , event , and comment fields (all the references). After that, I also deep-populate the event field to get event.creator . Here is my code for the query:

var activityPopulateObj = [
                { path: 'event' },
                { path: 'event.creator' },
                { path: 'comment' },
                { path: 'actor' },
                { path: 'recipient' },
                { path: 'event.creator' }
            ],
            eventPopulateObj = {
                path: 'event.creator',
                model: User
            };

Activity.find({ $or: [{recipient: user._id}, {actor: {$in: user.subscriptions}}, {event: {$in: user.attending}}], actor: { $ne: user._id} })
            .sort({ _id: -1 })
            .populate(activityPopulateObj)
            .exec(function(err, retrievedActivities) {
                if(err || !retrievedActivities) {
                    deferred.reject(new Error("No events found."));
                }
                else {
                    User.populate(retrievedActivities, eventPopulateObj, function(err, data){
                        if(err) {
                            deferred.reject(err.message);
                        }
                        else {
                            deferred.resolve(retrievedActivities);
                        }
                    });
                }
            });

This is already a relatively complex query, but I need to do even more. If it hits the part of the $or statement that says {actor: {$in: user.subscriptions}} , I also need to make sure that the event 's privacy field is equal to the string public . I tried using $elemMatch , but since the event has to be populated first, I couldn't query any of its fields. I need to achieve this same goal in multiple other queries, as well.

Is there any way for me to achieve this further filtering like I have described?

Answer 1

The answer is to change your schema.

You've fallen into the trap that many devs have before you when coming into document database development from a history of using relational databases: MongoDB is not a relational database and should not be treated like one.

You need to stop thinking about foreign keys and perfectly normalized data and instead, keep each document as self-contained as possible, thinking about how to best embed relevant associated data within your documents.

This doesn't mean you can't maintain associations as well. It might mean a structure like this, where you embed only necessary details, and query for the full record when needed:

var activitySchema = new mongoose.Schema({
  event: {
    _id: { type: ObjectId, ref: "Event" },
    name: String,
    private: String
  },

  // ... other fields
});

Rethinking your embed strategy will greatly simplify your queries and keep the query count to a minimum. populate will blow your count up quickly, and as your dataset grows this will very likely become a problem.

Answer 2

but since the event has to be populated first, I couldn't query any of its fields.

No can do. Mongodb cannot do joins. When you make a query, you can work with exactly one collection at a time. And FYI all those mongoose populates are additional, distinct database queries to load those records.

I don't have time to dive into the details of your schema and application, but most likely you will need to denormalize your data and store a copy of whatever event fields you need to join on in the primary collection.

Answer 3

You can try below aggregation. Look at this answer: https://stackoverflow.com/a/49329687/12729769

And then, you can use fields from $addFields in your query. Like

{score: {$gte: 5}}