Get result based on property of another document from one document in MongoDB

Question

I have two documents -

const taskSchema = new mongoose.Schema({
    leadId: { type: String, default: '' },
    customer_name: { type: String, default: '' }
})

const leadSchema = new mongoose.Schema({
  Id: { type: String, default: "" },
  associate_status: { type: Boolean, default: false },
})

I want to run a query on tasks such that I get all the leads where associate status is true.

LeadId in task schema is the same as Id in the lead schema.

I tried this -

  const report = await taskModel.aggregate([
   { $lookup: {
      from: 'leads',
      let: { leadId: '$leadId' },
      pipeline: [
        {
          $match: {
            $expr: {
              $eq: ['$Id', '$$leadId'],
            },
          },
        },
      ],
      as: 'leaditems',
    },
  }
  { $unwind: '$leaditems' },
  { $match: { leaditems.associate_status: true}}
 ])

But since I have a large number of documents (more than 200k leads and 100k tasks, and these numbers will keep on growing), the query sometimes runs after 9 to 10 seconds and sometimes doesn't runs at all. I am looking for a more optimized approach or any other implementation that will make this work.

Update:

I tried this as well but no improvement -

const report = await taskModel.aggregate([
       { $lookup: {
          from: 'leads',
          let: { leadId: '$leadId' },
          pipeline: [
            {
              $match: {
                $expr: {
                  $and: [
                   {
                     $eq: ['$Id', '$$leadId'],
                   },
                   {
                     $eq: ['$associate_status', true],
                   },
                 ],
               },
              },
            },
          ],
          as: 'leaditems',
        },
      }
     ])

Answer 1

Put $match condition in $lookup pipeline or even try to put it before $lookup .

Set index on both associated id.

Maybe your service has a 10 second database connection constraint, try to make it longer.

Answer 2

To use the index in $exp inside a lookup pipeline you need MongoDB 5. MongoDB 4.4.9 will not use the index and query will be slow.

• create the indexes in leads and Id, its expected to be much faster
  (index helps if it has good selectivity, if select small percentage of the documents, an index that will return 90% of documents can make things worse)(if you add many values on leadId index will be updated also, it can cost, but the benefits most of the times are much more)
• replace with a simple equality $lookup
• Also if you dont really need the $unwind and you prefer your data in array you can use $filter

const report = await taskModel.aggregate([
       { $lookup: {
          from: 'leads',
          localField: "leadId",
          foreignField: "Id",
          as : "leaditems"
       },
       { $unwind: '$leaditems' },
       { $match: { leaditems.associate_status: true}}       
     ])

If possible benchmark it after you do those and send some feedback.