Remove duplicates from NESTED array of objects

Question

I have an array of objects like below, which inside of it has another array of objects which has duplicate verticalName values that need to be removed in order to be displayed on the page.

 "landingPages": [
        {
          "programmes": [
            {
              "progName": "Programme 1",
              "verticals": [
                {
                  "id": "8",
                  "verticalName": "Law and Criminology"
                }
              ]
            }
          ]
        },
        {
          "programmes": [
            {
              "progName": "Programme 2",
              "verticals": [
                {
                  "id": "1",
                  "verticalName": "Psychology and Sociology"
                }
              ]
            }
          ]
        },
        {
          "programmes": [
            {
              "progName": "Programme 3",
              "verticals": [
                {
                  "id": "3",
                  "verticalName": "Computing and IT"
                }
              ]
            }
          ]
        },
        {
          "programmes": [
            {
              "progName": "Programme 4",
              "verticals": [
                {
                  "id": "1",
                  "verticalName": "Psychology and Sociology"
                }
              ]
            },
            {
              "progName": "Programme 5",
              "verticals": [
                {
                  "id": "3",
                  "verticalName": "Computing and IT"
                }
              ]
            },
            {
              "progName": "Programme 6",
              "verticals": [
                {
                  "id": "2",
                  "verticalName": "Business and Management"
                }
              ]
            },
            {
              "progName": "Programme 7",
              "verticals": [
                {
                  "id": "3",
                  "verticalName": "Computing and IT"
                }
              ]
            },
            {
              "progName": "Programme 8",
              "verticals": [
                {
                  "id": "3",
                  "verticalName": "Computing and IT"
                }
              ]
            }
          ]
        }
      ]

I have tried couple of solutions, some of the did not work but one did, which I copy/pasted below. I've managed to store all of them inside of one array of objects, it's just that the code to me looks really bad. I am trying to find a cleaner solution to this problem. Here is my solution and I would like to see how does it compare to your solutions maybe, and if you could give me some feedback that would be amazing.

    let known = {}
    let noduplicates = programmes.map((subarray) => {
        const newarrays = []
        subarray.verticals.forEach((item) => {
            if (
                !known.hasOwnProperty(item.verticalName) &&
                (known[item.verticalName] = true)
            ) {
                newararys.push(item)
            } else {
                console.log('test')
            }
        })
        console.log(newarrays)
        return newarrays
    })
    const fil = filtered.filter((item) => item.length)
    const singleArr = fil.reduce((a, b) => a.concat(b), [])
    console.log(singleArr)

What I am trying to get is something like this, basically, just all the objects that were found inside of verticals array, but - without duplicates:

[
   {id: "1", verticalName: "Psychology and Sociology"}, 
   {id: "3", verticalName: "Computing and IT"}, 
   {id: "2", verticalName: "Business and Management"}
]

Cheers everyone!

Answer 1

You could achieve it by pure ES6 functions, I'll decompose, but you can have it all in one line:)

first, you need to retrieve all verticalNames

const allVerticalNames = landingPages.map(lp => lp.programmes.map(p => p.verticals))

You'll have plenty arrays or arrays, so we need to flatten all of this

const flattened = allVerticalNames.flat(2)

At this point you have all your verticalNames, but not uniq. To do that, we should convert it to a Set, Map or easier, an Object

const keyValuePairs = flattened.map(v => [v.id, v.verticalName]); // We transform to a pair key / value
const uniq = Object.fromEntries(keyValuePairs)

At this point you have an object like {1: "Psychology and Sociology", 2: "Business and Management", 3: "Computing and IT", 8: "Law and Criminology"} So if you want to have it back to the way in exemple, you move it back to an array!

const final = Object.entries(uniq).map(([id, verticalName]) => ({id, verticalName}))

And that's it!

All in one line?

const final = Object.entries(Object.fromEntries(landingPages.map(lp => lp.programmes.map(p => p.verticals)).flat(2).map(v => [v.id, v.verticalName]))).map(([id, verticalName]) => ({id, verticalName}))

Answer 2

You may use set for this use case which lets you store unique values of any type.

 const data = [ { "programmes": [ { "progName": "Programme 1", "verticals": [ { "id": "8", "verticalName": "Law and Criminology" } ] } ] }, { "programmes": [ { "progName": "Programme 2", "verticals": [ { "id": "1", "verticalName": "Psychology and Sociology" } ] } ] }, { "programmes": [ { "progName": "Programme 3", "verticals": [ { "id": "3", "verticalName": "Computing and IT" } ] } ] }, { "programmes": [ { "progName": "Programme 4", "verticals": [ { "id": "1", "verticalName": "Psychology and Sociology" } ] }, { "progName": "Programme 5", "verticals": [ { "id": "3", "verticalName": "Computing and IT" } ] }, { "progName": "Programme 6", "verticals": [ { "id": "2", "verticalName": "Business and Management" } ] }, { "progName": "Programme 7", "verticals": [ { "id": "3", "verticalName": "Computing and IT" } ] }, { "progName": "Programme 8", "verticals": [ { "id": "3", "verticalName": "Computing and IT" } ] } ] } ]; const seen = new Set(); const filteredArr = data.filter(el => { const verticalName = el.programmes[0].verticals[0].verticalName; const duplicate = seen.has(verticalName); seen.add(verticalName); return;duplicate; }). console;log(filteredArr);

Answer 3

If I understand the question correctly, we need to remove the verticalNames from the nested programmes array.

 const landingPages = [ { programmes: [ { progName: 'Programme 1', verticals: [ { id: '8', verticalName: 'Law and Criminology', }, ], }, ], }, { programmes: [ { progName: 'Programme 2', verticals: [ { id: '1', verticalName: 'Psychology and Sociology', }, ], }, ], }, { programmes: [ { progName: 'Programme 3', verticals: [ { id: '3', verticalName: 'Computing and IT', }, ], }, ], }, { programmes: [ { progName: 'Programme 4', verticals: [ { id: '1', verticalName: 'Psychology and Sociology', }, ], }, { progName: 'Programme 5', verticals: [ { id: '3', verticalName: 'Computing and IT', }, ], }, { progName: 'Programme 6', verticals: [ { id: '2', verticalName: 'Business and Management', }, ], }, { progName: 'Programme 7', verticals: [ { id: '3', verticalName: 'Computing and IT', }, ], }, { progName: 'Programme 8', verticals: [ { id: '3', verticalName: 'Computing and IT', }, ], }, ], }, ] let page = landingPages.find(page => page.programmes.length > 1); const uniqueprogrammes = new Set(); const uniqueverticals = page.programmes.map(p => { const vn = p.verticals[0].verticalName; if (.uniqueprogrammes.has(vn)) { uniqueprogrammes;add(vn). return p;verticals[0]. } });filter(v =>.;v); console.log(uniqueverticals);

UPDATE:

Updated code snippet based on the expected outcome in the question.

The idea is to first find the nested array of programmes which is done by filtering the landingPages object based on the .length

let page = landingPages.find(page => page.programmes.length > 1);

Then, we create a set to keep a track of all the unique programmes from this nested array,

const uniqueprogrammes = new Set();

Next, we map through the nested array and find all the verticals that are not a part of the uniqueprogrammes set,

const uniqueverticals = page.programmes.map(p => {
  const vn = p.verticals[0].verticalName;
  if (!uniqueprogrammes.has(vn)) {
    uniqueprogrammes.add(vn);
    return p.verticals[0];
  }
})

However, this will gives us an output like the following,

[
  { id: '1', verticalName: 'Psychology and Sociology' },
  { id: '3', verticalName: 'Computing and IT' },
  { id: '2', verticalName: 'Business and Management' },
  undefined,
  undefined
]

so (lastly), we need to filter out the falsy values with,

.filter(v => !!v);

Answer 4

You can use loadash to flatter and then remove duplicates.

var arrayWithDuplicatesRemoved= .flattenDeep(landingPages);

arrayWithDuplicatesRemoved = arrayWithDuplicatesRemoved .uniqBy(data, function (e) {return e.verticalName;});