node.js : how to make express app communicate with kue running on different cpu of same box?

Question

My app will need to do some cpu intensive processing, so I'm thinking of doing that as a background job.

I want both the express app and kue to run on the same box (different cores). How do I do that? And how do the main app and the job queue communicate with each other? They're on the same box, so sending out an http request seems very wasteful.

Answer 1

You could start multiple Node processes (your Express server and your background task handler) manually, or through a process manager like pm2 or forever ; or use child_process.fork from your Express app to start up a new Node process that will run your CPU-intensive jobs.

There isn't a 100% guarantee that both will run on separate cores, but you can let your OS do all that scheduling and it will most likely do the right thing.

As for Kue: it uses Redis as a go-between to store jobs. Your Express app posts jobs to it, and your background task will take each job off of it, process it, and optionally post a message back.