What is a more machine efficient way than for loop + lodash combo?

Question

Update: Edited the code for a clearer explanation.

I need to find x number of object in an array against another array of properties, I will use a for loop + lodash.filter to retrieve the list. I was wondering how can it be more efficient for NodeJs since NodeJs is single threaded. An example use case is to reduce the number of mysql calls. For example, I have a list of products and i want to retrieve a subset via their product category. So I will get something like this:

const lodash = require('lodash');
...
let productCategories = [{id:111,...},{id:222,...},{id:333,...},{id:444,...}];
let products = await this.getProductByProductCategories(lodash.map(productCategories,'id')); //Does mysql call whereIn ... which I will not include here.

for(let productCategory of productCategories){
    let productSubset = lodash.filter(products, {product_category_id: productCategory.id};
     //Do something with productSubset
     //I also want to reference the parent object here: productCategory.xxx
}

Is there any more efficient (and code friendly) way of doing the for loop + lodash.filter combo?

Answer 1

There's no need to do filter inside for loop. You need to use _.groupBy instead.

Eg

...
const productSubsets = _.groupBy(products, 'product_category_id');
for (let key in productSubsets) {
  // do something with productSubsets[key]
}

Answer 2

Not sure what lodash does under the hood, but a nested for loop like this typically takes O(n^2) time; however, using hash maps, you can reduce this to O(n) time, eg

const productCategoryIds = [111,222,333,444,555];
const mappedProductCategoryIds = productCategoryIds.reduce((obj, id) => {
    obj[id] = true;
    return obj;
}, {});

const products = await this.getProductByProductCategories(productCategoryIds); //Does mysql call whereIn ... which I will not include here.

const productSubset = products.filter(product => mappedProductCategoryIds[product.product_category_id]);

//Do something with productSubset

We first iterate through the productCategoryIds and create a hash map ( mappedProductCategoryIds ), against which we can test the product_category_id of any product simply by checking to see if the value at key product_category_id in that hash map mappedProductCategoryIds is truthy (ie whether it is true ) or falsey (ie undefined ). (A hash map lookup takes O(1) time.)

JavaScript filter() will collect the product if mappedProductCategoryIds contains its product_category_id ; otherwise, it will reject it.

Answer 3

If you wanna stick to lodash you can use _.groupBy and later do something with values of produced object:

const productsByCategoryId = _.groupBy(products, 'product_category_id');

You can retrieve values (eg with lodash's _.values()) or map it with _.mapValues() or just use plain javascript to access data you need.

It's more elegant but probably not really faster. If you want to be more efficient in asynchronous environment maybe you should try going reactive like:

from(products)
  .pipe(
    groupBy(product => product.product_category_id),
    //do something in pipe 
  )

More info: https://www.learnrxjs.io/operators/transformation/groupby.html

Answer 4

Depends on how you want to use subset exactly, but you could consider returning an ordered set of products from the DB

function doSomethingWithProductSubset(productSubset) {
    // do something with product subset
    console.dir(productSubset);
}
const productCategoryIds = [111,222,333,444,555];
const products = await getProductByProductCategoriesOrderedByCategory(productCategoryIds);
if (products && products.length) {
    let productSubset = [];
    let category = products[0].category;
    products.forEach(product => {
        if (productSubset.length && product.category != category) {
            doSomethingWithProductSubset(productSubset);
            productSubset = [];
            category = product.category;
        }
        productSubset.push(product);
    });
    if (productSubset.length) doSomethingWithProductSubset(productSubset);
}