Effecient Javascript Object filtering using multiple array indexes. How much data before using Node?

Question

I'm looking for both practical, and also theoretical insight about my application.

Take 50,000 js objects, with 5 properties each, structured as

0: Object
  CostCenter: "1174"
  Country: "USA"
  Job: "110-Article Search"
  Team: "Financial"
  Username: "anderson"

And take 5 respective arrays (one for each object property) such as the 'Country' array

4: Array[4]
  0: "Asia Pacific"
  1: "Australia"
  2: "Brazil"
  3: "Canada"

What is the most efficient way to filter the 50,000 objects, eliminating all objects which have at least one property which has 0 matches in its respective array.

The max sizes for the arrays are:

  CostCenter,  77
  Country,     27
  Job,         27
  Team,        10
  Username,    99

My first idea is to loop through the 50,000 objects, and

if the 'CostCenter' property === any CostCenter array item,
push the object into a temporary array of objects 

Which might leave me with only 20,000 objects in the temporary array. Then repeat this process for each property and its respective filtering array, building a new temporary object each time.

Finally this process would leave me with the last array, which would be the resultant data after going through 5 filters.

It takes about 20 seconds for me to download the 18mb JSON file (but I'm okay with that)

...which is exponentially longer than the time it takes my chrome browser on 16gb ram to process the JSON into 50,000 js objects AND loop these objects to dynamically build the filtering arrays with all unique values contained in the JSON.

Is this efficient? It seems very very fast for the amount of data being processed, but I also get the feeling some user environments (like my boss' iPad) may run out of in-browser memory.

What better ways are there?

Should I do this in Node.JS? I am a javascript programmer, so that seems like it may not take too long to learn. Plus Node is super duper hip these days... maybe I should get on with it.

Will some browsers fail to download an 18mb json file? Where can I find info about limits?

Answer 1

Basically you want

var arrays = {
    "Country": […],
    …
};
var result = my50000items.filter(function(item) {
    for (var prop in arrays)
        if (arrays[prop].indexOf(item[prop]) == -1)
             return false;
    return true;
});

You can optimise this by replacing the indexOf call with a faster property lookup. To do so, make:

var lookups = {};
for (var prop in arrays) {
    var obj = lookups[prop] = {};
    for (var i=0; i<arrays[prop].length; i++)
        obj[arrays[prop][i]] = true;
}

Then you can use

var result = my50000items.filter(function(item) {
    for (var prop in lookups)
        if (!lookups[prop][item[prop]])
             return false;
    return true;
});