How to perform array join in Node.js efficiently/fast similar to MongoDB $lookup?

Question

I want to perform a $lookup in Node.js similar to $lookup aggreation from MongoDB.

I have a solution but I'm not sure how fast it performs with more objects in each of the two arrays or with bigger objects.

let users = [
    {userId: 1, name: 'Mike'}, 
    {userId: 2, name: 'John'}
    ]
let comments = [
    {userId: 1, text: 'Hello'}, 
    {userId: 1, text: 'Hi'}, 
    {userId: 2, text: 'Hello'}
    ]

let commentsUsers = [
    {userId: 1, text: 'Hello', user: {userId: 1, name: 'Mike'}}, 
    {userId: 1, text: 'Hi', user: {userId: 1, name: 'Mike'}}, 
    {userId: 2, text: 'Hello', user: {userId: 2, name: 'John'}}
    ] //Desired result

I know this can be done easily with ECMA6 arrays. For example:

let commentsUsers = comments.map(comment => {comment, users.find(user => user.userId === comment.userId)} )

I that an effective way to do this for a large number of users eg. 1M users. How does lodash compare to this or any other more specialized library? Are there better ways to do this with vanilla JS eg. with Array.prototype.reduce()? Can indexing be used in any way to improve the performance of the join?

Edit:

My ideal solution

 let users = [{userId:1,name:'Mike'},{userId:2,name:'John'}] let comments = [{userId:1,text:'Hello'},{userId:1,text:'Hi'},{userId:2,text:'Hello'}]; let usersMap = new Map(users.map(user => [user.userId, user])) let commentsUsers = comments.map(comment => ({...comment, user: usersMap.get(comment.userId)})) console.log(commentsUsers) 

Thanks for the feedback!

Answer 1

Your desired result is not a proper data structure. You are missing a key to your object of eg {userId: 1, name: 'Mike'} . I added user as the key value for a indexing solution.

First I create a Map where the userId will be our loop-up value. Afterwards I just iterate over the comments with map , transforming each object to a new one that contains all the comment information plus a new kv pair of user. For that pair we don't need to use find anymore instead we have a simple HashMap get call.

Time-complexity-wise this changes the code from O(n^2) to O(n) .

 let users = [{userId:1,name:'Mike'},{userId:2,name:'John'}], comments = [{userId:1,text:'Hello'},{userId:1,text:'Hi'},{userId:2,text:'Hello'}]; function mergeCommentUser(users, comments) { let map = new Map(users.map(v => [v.userId, v])); return comments.map(o => ({...o, user: map.get(o.userId)})); } console.log(JSON.stringify(mergeCommentUser(users,comments))) 

Depending on what you want (and to save on redundancy), you could also change the following line:

let map = new Map(users.map(v => [v.userId, v]));

to the following instead:

let map = new Map(users.map(v => [v.userId, v.name]));

By that your result would look like:

[
    {"userId":1,"text":"Hello","user":"Mike"},
    {"userId":1,"text":"Hi","user":"Mike"},
    {"userId":2,"text":"Hello","user":"Paul"}
]

Otherwise, you could omit the comment.userId and instead add the full user to the object for another way to avoid redundancy.

Answer 2

Currently, the code example you provide is O(n * m), or, O(n ² ). You could create a map of each of the userId's and their respective indexes in the users array, and then rather than find the user, you can directly access it by index. This will reduce the time to O(n + m), that is, O(n).

The code would look something like this:

 const users = [{ userId: 1, name: "Mike" }, { userId: 2, name: "John" }]; const comments = [ { userId: 1, text: "Hello" }, { userId: 1, text: "Hi" }, { userId: 2, text: "Hello" } ]; const map = new Map(users.map((o, i) => [o.userId, i])); console.log( comments.map(o => { const index = map.get(o.userId); return index !== undefined ? { comment: o.text, user: users[index] } : o; }) ); 

Obviously, you can modify the end result, but this approach would be much more efficient than the one you proposed.