MongoDb slow aggregation with many collections (lookup)

Question

i'm working on a MEAN stack project, i use too many collections in my aggregation so i use a lot of lookup, and that impacts negatively the performance and makes the execution of aggregation very slow. i was wondering if you have any suggestions, i found that we can reduce lookup by creating for each collection i need an array of objects into a globale collection however, i'm looking for an optimale and secured solution. As an information, i defined indexes on all collections into mongo.

Thanks for sharing your ideas!

Answer 1

This is a very involved question. Even if you gave all your schemas and queries, it would take too long to answer, and be very specific to your case (ie. not useful to anyone else coming along later).

Instead for a general answer, I'd advise you to read into denormalization and consider some database redesign if this query is core to your project.

Here is a good article to get you started.

Denormalization allows you to avoid some application-level joins, at the expense of having more complex and expensive updates. Denormalizing one or more fields makes sense if those fields are read much more often than they are updated.

A simple example to outline it:

Say you have a Blog with a comment collection, and a user collection You want to display the comment with the name of the user. So you have to load the player for every comment. Instead you could save the username on the comment collection as well as the user collection. Then you will have a fast query to show comments, as you don't need to load the users too. But if the user changes their name, then you will have to update all of the comments with the new name. This is the main tradeoff.

If a DB redesign is too difficult, I suggest splitting into multiple aggregates and combining them in memory (ie. in your node server side code)