executing a code of webservice in another child process in node.js

Question

I need an understanding and code architecture of how to run a time consuming web service in a separate child process. Is it good to run in a separate child process such kind of web service which will run longer. The web service will receive a pay load of upto like 2000 nested json objects, numbers could be less or more, and it will fetch the data and insert into the database. Below is just the code snippet just to give an idea.

router.post('/longRunningWebservice', function (req, res) {
    //for loop to iterate 2000 json objects 
    //it has some queries to insert data
    //it has some if conditions to check
    //and finally res.send()
}); 

I need to understand and want to run the the code in a separate child process to make it more scalable, and also utilize the other cores.

Answer 1

1. fork some workers at the begining of the code
2. in your code write seperate worker tasks and master tasks 'if (cluster.isMaster) {...} else {...}'
3. send the data which needs to be processed to the worker with IPC messaging or redis pub/sub messaging, after the worker processed the data it can send it back to the master which handles the response.

other notes:

• you can set multiple workers and determine which is busy and which is not
• redis pub/sub if better then IPC cuz with IPC you can only communicate with master - worker, with redis pub/sub you can communicate with master - worker and worker - worker.
• best practice is to setup a master which only forks workers and refork them if they die cuz of an error or something, you can also create 1 worker which handles http requests and 3 workers which processes data eg. if you have 4 processor cores.

additional approach: Fork workers with all the same tasks (handle http and data processing), cluster will automatically handle load ballance and instead of for loop create a recursive function which process a single json object then call itself with setimmediate so the data processing will not block event loop and a single worker can handle multiple requests. easy way