从 typescript 中的对象列表中查找键的唯一值的高效方法

Question

I am working on an angular app that has a

1. a set of filters
2. and records present in a table

Columns in the table correspond to filters. At any time filters contain unique values from corresponding columns as options.

For a record, a column can contain more than 1 values (ie more than 1 options from the corresponding filter)

When a user selects an option from a filter, the records in the table are filtered and the filtered results (as per user the selection) are shown to the user.

Once the set of filtered records is derived, unique values for each filter are derived from the set of filtered records by finding unique values for each column.

Key of Filter objects correspond to columns of Record objects I have a list of records and a list of filters. I want to iterate over both these lists and find the unique value of columns for each key.

I am using the below logic to find unique options for filters from my messages.

export function filterOptionsService(records: Record[], filters: RecordFilter[]): RecordFilter[] {
    const newFilters: RecordFilter[] = filters.map(filter => {

        //logic to find all values for a column from the set of records 
        const filterOptions = records.reduce((uniqueList, record) => uniqueList.concat(record[filter.key]), []);

        //logic to find unique values for a column from the set of records
        //which act as options of corresponding filters.
        const uniqueOptions = uniqBy(filterOptions, (opt) => filter.valueFunction ? filter.valueFunction(opt) : opt);

        const filterOptions: FilterOption[] = uniqueOptions.map(value => {
            return {
                label: filter.labelFunction ? filter.labelFunction(value) : value,
                value: filter.valueFunction ? filter.valueFunction(value) : value,
            };
        });
        filter.options = orderBy(dropListOptions, 'label');

        //here is my logic to find the count of each option, present in the filtered records
        filter.options = filter.options.map(option => ({
            ...option,
            count: filter.valueFunction
                ? filterOptions.filter(value => filter.valueFunction(value) === option.value).length
                : filterOptions.filter(value => value === option.value).length
        }));
        return filter;
    });
    return newFilters;
}

interface Filter {
    key: string;
    labelFunction?: Function;
    valueFunction?: Function;
    order: number;
    multiple: boolean;
    options: DropListOption[];
    values: DropListOption[];
}


interface Record {
    column1: string;
    column2: string;
    column3: string | string[];
    .
    .
    columnN: string;
}

The below logic takes the most time in my code. It takes around 7 seconds for 8k records.

const filterOptions = records.reduce((uniqueList, record) => 
                              uniqueList.concat(record[filter.key]), []); 

I am unable to make my code perform better. Please can you suggest where am I going wrong.

Here is sample code on typescrip playground https://www.typescriptlang.org/play?#code/FASwdgLgpgTgZgQwMZQAQDEQBtowPIAOEIA9mKgN7Co201YIBGUWAXKgM4QzgDmA3NTq0AbgiwBXKOwRgAnoOG1kxEVAD87RiRJYosxUtQEEcrCQQATTalkKhSpCQmQbYCQFtmMQzQC+-LTA1AD0IRwIHgR6qGJ6HKhwJDCoANZQcrYwaAQwJASwEHIAjAA0xnkFMEUATKigkLCIKKgASlBOMJaYOLCUDsLpcuxcPGAC9UaoDMxY6C5IxGQ282CLpGC+wnFSq+vL7HtLmwN0yZaw7O5esFt0HhI4INHSqNq6+idT+ccch9i4QjHADaAF07qJxFI-hgAbAgRswVsAkFgGEKvlqiAoAkSHA2h1zqgkAwOAkENlOJE0AgEkNUHiCZ1unCUiTaRw0SFUBAABYgBICzggKJYEBwbGWVDZKo4yAIY6oDyZFQScTE0kJXm0pV2JXJHKVQrYzngXDNNCtEgAd36dFy+UKJRG3D4SIGDqqtRdYwEwACwWATjAXFQ4BAxHEVutMOjYNQAF5UMDTlNhBRPU7iqxgQAiMhQXPlXN87KF0GlTPVOQ1Vi5phIXN+UqptO0DNG6vZvMStRF1C5sDgcuVzve3MXOBNlttqYdx1dnP5sCF4ulqCrge9zeD4e5itV8cNputtPg4JBMIASVsHgZYCwKrJIF4Q-GsShOJ5JDSGQxXsyJIUg4EVonFEAkAVDYgzIUNw0jOZWVjQkuh6XB4yTFNZyUCghjrQ8Sn7c5LjKJVHmIF5WEQLAOCgcofg2P4wXKHYcXYMFm1PNNcIyfCxxrIiukuGpygeJ5KOo2j6KIRicwrD9JDY5NQT8GguKMc9gglXp8Bk2CAGVYBECCoAACngkAoxtDhSgs8Q0NgDgAE pBGCdEb21NRUGKAAWRJWQZRgACsOggTl0WtXkoHIDyEC8gAOVJpRtQKQsWBI+TQWYUuySwJBaTEICgsgeQQdIEgANgAWgAdk4QkwEsDgADpgDgBZFW0wE9JDQyYGMlBTLyGN2DjeSusc0aUJZHSwSc-pTgmmAWo8BACFMpbEwAPjtIwwnMXgIO-fzGtsLAsAU6FEmSWxiV0TxyDgPI70y+qIAZfFsmZBJT2DUMloRWDE2SmNmty-KzNMlwQAARykAAZAUIHKYb5oTHbUGhuGoERrhmuDSCICGm1gSW5qhlBJzyjmwIuN+2CPmag7TIAcgBnqOBZ0p2d+FzsNbPwXP9QMgA

Answer 1

I think the performance problem you're having is that Array.prototype.concat() does not modify an existing array, but instead returns a new array. Immutability is nice, but it doesn't seem relevant to your use case: every uniqueList array you create except for the very last one will be discarded. Object creation is fairly fast in JavaScript, but creating thousands of array objects only to immediately throw them away is slowing things down.

My suggestion would be to replace concat() with something that modifies the existing array, such as Array.prototype.push() :

That is, you could change

const filterOptions = rows.reduce(
  (uniqueList, row) => uniqueList.concat(row[filter.key]), []
);

to

const filterOptions: string[] = [];
for (let row of rows) filterOptions.push(...row[filter.key]);

When I run a simulation where I create 6000 rows and 14 filters (see playground link below), the concat() version takes about 7.5 seconds, whereas the push() version takes about 38 milliseconds . Hopefully that factor of ~200 improvement holds in your environment and makes enough of an impact to be sufficient for your needs.

Note that I'm not dealing with any "uniquifying" or "function" aspects that your original problem seems to have, since your reproducible example code doesn't touch that either. Unique strings might be more easily tallied via an object key than an array, or maybe even via a Set . But again, hopefully changing concat() to push() will be enough for you.

Playground link to code