mongodb - unwinding nested subdocuments

Question

For the following dataset example:

lists

{ _id: 1, included_lists: [ 2 ], items: [ "i1" ]}
{ _id: 2, included_lists: [], items: [ "i2", "i3" ]}

items

{ _id: "i1", details: [{}, {}, {}]}
{ _id: "i2", details: [{}, {}, {}]}
{ _id: "i3", details: [{}, {}, {}]}

I want to grab all the items for a list, including the ones attached to the included_lists

For example: if we're looking at list _id 1, we should get items i1, i2, i3

I have an idea how to do this, which involves using populate or $lookup , but I'm not sure how to unwind the nested items inside the included_lists and join them with the items in the original list.

In the end, I would like to have a dataset where I am able to use limit , skip and match .

I'm using mongoose, but vanilla mongodb code would also be fine.

Update

My current idea of how to do this is to retrieve all of the list ids first in one query ie

List.find({ _id: id}, { included_lists: 1})

Then, with the list ids, make an array of that ie

var all_ids = [id, ...included_lists]

Then just find the items and unwind

Psuedo-code:

List
    .aggregate([
        {
            $match: {
                _id: {
                    $in: all_ids
                }
            }
        },
        { $lookup: {} }
        {
            $unwind: "$items"
        },
        {
            $project: {
                "list.name": 1,
                "list._id": 1,
                "items": 1
            }
        }
    ])

But I don't want to have to do a first query to retrieve all the list_ids, I should be able to retrieve all related items just through one _id which would then be able to retrieve the rest of the items through included_lists

Answer 1

You can try below aggregation from mongodb 3.6 and above

List.aggregate([
  { "$match": { "_id": id }},
  { "$lookup": {
    "from": Items.collection.name,
    "let": { "items": "$items" },
    "pipeline": [
      { "$match": { "$expr": { "$in": [ "$_id", "$$items" ] } } }
    ],
    "as": "items"
  }},
  { "$lookup": {
    "from": Lists.collection.name,
    "let": { "included_lists": "$included_lists", "items": "$items" },
    "pipeline": [
      { "$match": { "$expr": { "$in": [ "$_id", "$$included_lists" ] } } },
      { "$lookup": {
        "from": Items.collection.name,
        "let": { "items": "$items" },
        "pipeline": [
          { "$match": { "$expr": { "$in": [ "$_id", "$$items" ] } } }
        ],
        "as": "items"
      }},
      { "$project": { "allItems": { "$concatArrays": [ "$$items", "$items" ]}}}
    ],
    "as": "included_lists"
  }},
  { "$unwind": "$included_lists" },
  { "$replaceRoot": { "newRoot": "$included_lists" }}
])

Answer 2

You can try below aggregation in 3.4.

Initial $lookup to get the items values for included_lists followed by $concatArrays to merge the looked up items and items.

Second $lookup to get the item details followed by $unwind to flatten the results.

List.aggregate([
{"$lookup":{
  "from":name of the list collection,
  "localField":"included_lists",
  "foreignField":"_id",
  "as":"included_items"
}},
{"$unwind":"$included_items"},
{"$project":{"allItems":{"$concatArrays":["$items","$included_items.items"]}}},
{"$lookup":{
  "from":name of the item collection,
  "localField":"allItems",
  "foreignField":"_id",
  "as":"lookedup_items"
}},
{"$unwind":"$lookedup_items"},
{"$skip": some number},
{"$limit": some number}
])