JavaScript Array.filter uniqes and not block the UI

Question

I have a specific use case where some validation logic has to happen in the UI (for various business reasons[...]). The array may contain up to several tens or up to a few hundred thousand items (1-400K). The frontend is Angular based.

The first step is to check for duplicates (and store them in another array[...]). This is accomplished with below:

validateTargets(targets: string[]): ValidationResultObject[] {

    let result: ValidationResultObject[];
    let dups: string[] = [];

    var uniques = targets.filter( (item,index) => {
        if (targets.indexOf(item) === index) {
        return targets.indexOf(item) === index
        }
        else {
            dups.push(targets[index])
        }
    }

    //other validation logic goes here

    return result;
}

Problem is an obvious UI freeze when this runs against anything above 50K. For the time being I've put above as callback in another function in a setTimeout to at least allow the UI to run a spinner while the page hangs :)

I've seen several ways people advise how to design code to allow UI to be responsive (or at least re-draw); however, my case is a bit tricky since I deal with duplicates.

I was thinking to break down the array to chunks and run above Array.filter part in a loop within a setTimeout (for UI) BUT I later need to compare the chunks against themselves anyways so it just prolongs the logic! I do not feel comfortable enough to experiment with workers as there are browsers in the organization that do not support those.

Does anyone have an idea how to work this out? No, it is not possible to move this to backend :(

Regards

Answer 1

You can filter out duplicates much, much more efficiently:

let filtered = targets.reduce((result, item) => {
  result[item] = 1;
  return result;
}, {});
let noDuplicates = Object.keys(filtered);

That makes one pass over the array, and leverages the internal efficiency of property name lookup over the sequential search of .indexOf() . For an initial array with the extremely large number of elements you have, this should run in a comparably tiny amount of time.

Answer 2

我从未遇到过这种情况，但我想问一下您是否在 javascript 中使用了Set类型，因为它在内部删除了重复项，并且比过滤JS Set vs Array更高效，如果您的浏览器不支持 Set，您仍然可以使用 polyfill。

Answer 3

You could use an asynchronous function to process this amount of data. When it's finished call a callback with the result as an argument to continue the normal flow after it is finished.

async validateTargets(targets: string[], callback: function): {
 //...Logic
 callback(result)
}

Also, to remove duplicates you could use

[...new Set(items)]

Note: This will only work if the Items array contains only primitive values