merge and remove elements in nested arrays

Question

i have this array, i want to merge all elements inside the objects in the nested arrays and remove the duplicates.. the array is the output of mongo db populate so answers from there or just js will be amazing :)

 "visitors": [
        [
            {
                "name": "matan",
                "id": "61793e6a0e08cdcaf213c0b1"
            },
            {
                "name": "shani",
                "id": "61793e910e08cdcaf213c0b5"
            }
        ],
        [
            {
                "name": "david",
                "id": "6179869cb4944c6b19b05a23"
            },
            {
                "name": "orit",
                "id": "617986e535fdf4942ef659bd"
            }
        ],
        [
            {
                "name": "david",
                "id": "6179869cb4944c6b19b05a23"
            },
            {
                "name": "orit",
                "id": "617986e535fdf4942ef659bd"
            }
        ]
    ]

would like this output -

"visitors": [
        {
            "name": "matan",
            "id": "61793e6a0e08cdcaf213c0b1"
        },
        {
            "name": "shani",
            "id": "61793e910e08cdcaf213c0b5"
        },
        {
            "name": "david",
            "id": "6179869cb4944c6b19b05a23"
        },
        {
            "name": "orit",
            "id": "617986e535fdf4942ef659bd"
        },
]

these are my collections i need to get all visitors on one solar system, so > solars > planets > visitors

const solarsModel = new Schema({
    planets: [ { type: Schema.Types.ObjectId ,ref:'planet'} ],
    starName: { type: String, required: true, default: "" }
})

const planetModel = new Schema({
    planetName: { type: String, required: true, default: "" },
    system:{type: Schema.Types.ObjectId, ref: 'solar'},
    visitors: [{ type: Schema.Types.ObjectId , ref: 'visitor'}]
})

const visitorModel = new Schema({
    visitorName:{ type: String, required: true, default: "" },
    homePlanet: {type: Schema.Types.ObjectId, ref:"planet" },
    visitedPlanets: [{ type: Schema.Types.ObjectId, ref:"planet" }]
})

this is what i did to achieve a result would love to use Aggregate..

  const response = await solarModel
    .findById({ _id: data.id })
    .select({ starName: 1, _id: 0 })
    .populate({
      path: "planets",
      select: { visitors: 1, _id: 0 },
      populate: {
        path: "visitors",
        select: "visitorName",
      },
    })
    .exec();

Answer 1

(1) Flatten the array of arrays

visitors = visitors.flat();

Which gives us this:

[
  { name: 'matan', id: '61793e6a0e08cdcaf213c0b1' },
  { name: 'shani', id: '61793e910e08cdcaf213c0b5' },
  { name: 'david', id: '6179869cb4944c6b19b05a23' },
  { name: 'orit', id: '617986e535fdf4942ef659bd' },
  { name: 'david', id: '6179869cb4944c6b19b05a23' },
  { name: 'orit', id: '617986e535fdf4942ef659bd' }
]

(2) Get unique ids

let uniqueIds= [...new Set(visitors.map(v => v.id)]

Which gives us this:

[
   '61793e6a0e08cdcaf213c0b1',
   '61793e910e08cdcaf213c0b5',
   '6179869cb4944c6b19b05a23',
   '617986e535fdf4942ef659bd'
]

(3) Get new list of visitors based only on uniqueIds

visitors = uniqueIds.map(id => {
   let name = visitors.find(v => v.id === id).name;

   return {
      id,
      name
   }
});

Which gives us this:

[
  { name: 'matan', id: '61793e6a0e08cdcaf213c0b1' },
  { name: 'shani', id: '61793e910e08cdcaf213c0b5' },
  { name: 'david', id: '6179869cb4944c6b19b05a23' },
  { name: 'orit', id: '617986e535fdf4942ef659bd' },
]

Answer 2

You can use aggregate() like this:

• $unwind twice due to nested array
• $group using $addToSet to not get duplicates.

db.collection.aggregate([
  {
    "$unwind": "$visitors"
  },
  {
    "$unwind": "$visitors"
  },
  {
    "$group": {
      "_id": null,
      "visitors": {
        "$addToSet": {
          "id": "$visitors.id",
          "name": "$visitors.name"
        }
      }
    }
  }
])

Example here

Answer 3

Query

• reduce with concat to flatten
• union with an empty array,just to remove duplicates
• if you have other fields except visitors they are not affected

*try it on your driver, playground has a problem with field orders sometimes it loses it, here we compare documents to remove the duplicates.

aggregate(
[{"$set": 
   {"visitors": 
     {"$setUnion": 
       [{"$reduce": 
         {"input": "$visitors",
          "initialValue": [],
          "in": {"$concatArrays": ["$$value", "$$this"]}}},
        []]}}}])

Results

[{
  "visitors": [
    {
      "name": "david",
      "id": "6179869cb4944c6b19b05a23"
    },
    {
      "name": "matan",
      "id": "61793e6a0e08cdcaf213c0b1"
    },
    {
      "name": "orit",
      "id": "617986e535fdf4942ef659bd"
    },
    {
      "name": "shani",
      "id": "61793e910e08cdcaf213c0b5"
    }
  ]
}]