Pass and access request response object in child process using fork

Question

I want to know how to pass request response object from parent process to child process using fork. I have done is

var child = cp.fork(__dirname + '/childProcess.js', req, res);
child.on('message', function(m) 
{
    console.log("child process returned data " + m);
});
child.send("hello");

childProcess.js

var req = process.argv[2];
var res = process.argv[3];
process.on('message', (msg) => 
{
    console.log("req object :-" + req );
    console.log("res object :-" + res);
}
process.send("callparent");

which gives me undefined in child process. I have tried child.send("hello", req.socket ) too. but then i cant access methods of req in child process. it shows as circular structure.

Answer 1

According to the docs :

child.send(message[, sendHandle][, callback])

The optional sendHandle argument that may be passed to child.send() is for passing a TCP server or socket object to the child process. The child will receive the object as the second argument passed to the callback function registered on the process.on('message') event.

This suggests that you should send the req or req.socket object in the child.send() function.

Also, you are not accessing the req or req.socket object from the second argument in the function registered to the process.on('message') event.

Answer 2

The other answers are telling you to send the socket to the child and have the child listen and process the requests, which is not the question that was asked (how to send req/res to child). If your goal is to pass the socket to the child and have the child listen and process requests, those answers are fine. If your goal is to listen for and pre-process requests in the main process, and pass those requests to a child process for further processing, see below:

The short answer is that you have to actually explicitly send the request and response to the child process (they will not just magically appear in the child process). But there are some challenges to that.

In our application we have a similar problem. Our main request processor inspects requests to see which child process should handle each request, then sends that request to the child process (via child.send() ).

The problem with passing the request/response between parent and child is that you have to be able to serialize the objects sent back and forth with send , and the request object in particular cannot be serialized (it's chock full of circular references, has references to many other objects, etc).

So what we do is pull the subset of data out of the request that the child process needs, and we send that "lightweight request" object to the child process via child.send() , while keeping a reference to the original request in a queue on the parent process. When the child process completes processing, it sends its response data back to the parent via parent.send() , and the parent matches that up with the original request in the pending queue, and satisfies the request using the response data.

The advantage to this approach (as opposed to just making a custom protocol between parent/child) is that the child request processing code is still operating on request/response objects (just lightweight versions of them). In our case it meant we could use the same request processing code whether we were calling it in a child process, or were letting the process listen and handle the requests directly.

For example, here is how to make a lightweight, serializable request object and send it to the child process:

var requestData = 
{
    httpVersion: request.httpVersion,
    httpVersionMajor: request.httpVersionMajor,
    httpVersionMinor: request.httpVersionMinor,
    method: request.method,
    url: request.url,
    headers: request.headers,
    body: request.body
}
child.send(requestData);

You could also just using something like lodash pick to grab the fields you want.

The tricky part with this solution is that if you need to call any methods on request/response, you have to do that in the parent (where the actual objects are - all the child has is the selected data).

Answer 3

You create a server, and send handle to the client. If you use express for example, you can do:

Server.js

child= require('child_process').fork('child.js');    
const server = require('net').createServer();
server.on('connection', function(socket)  {    
    child.send('handle', socket);

}).listen(80);

Then you create a child, and accept the handle:

Child.js

var app=require('express')()
app.get('/',function(req,res){
     console.log(req.headers)
     res.send('Working!')
})  

var myServer=require('http').createServer(app)
process.on('message', function(m, server)  {
  if (m === 'handle') myServer.listen(handle)
});

More Info:

• Server.listen can get a handle: https://nodejs.org/api/http.html#http_server_listen_handle_callback
• Sending an handle to child process: https://nodejs.org/api/child_process.html#child_process_child_send_message_sendhandle_callback

Answer 4

Probably this below example will help illustrate this better. This has a parent and a child, parent creates a server, and upon client connections, the socket handle is passed onto the child.

The child has only a message loop in which it receives the handle, and is able to reach to the client through that.

$ cat foo.js

const cp = require('child_process')
const n = require('net')

if (process.argv[2] === 'child') {
  process.on('message', (m, h) =>  {
    h.end('ok')
  })
}
else {
  const c = cp.fork(process.argv[1], ['child'])
  const s = n.createServer();
  s.on('connection', (h) => {
    c.send({}, h)
  }).listen(12000, () => {
  const m = n.connect(12000)
  m.on('data', (d) => {
    console.log(d.toString())
  })
 })
}

strace with relevant sections around the handshake.

[pid 12055] sendmsg(11, {msg_name(0)=NULL, msg_iov(1)=[{"{\"cmd\":\"NODE_HANDLE\",\"type\":\"net.Socket\",\"msg\":{},\"key\":\"6::::12000\"}\n", 70}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {14}}, msg_flags=0}, 0) = 70

[pid 12061] recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"{\"cmd\":\"NODE_HANDLE\",\"type\":\"net.Socket\",\"msg\":{},\"key\":\"6::::12000\"}\n", 65536}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {12}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 70

[pid 12061] getsockname(12, {sa_family=AF_INET6, sin6_port=htons(12000), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0

[pid 12061] getsockopt(12, SOL_SOCKET, SO_TYPE, [1], [4]) = 0

[pid 12061] write(3, "{\"cmd\":\"NODE_HANDLE_ACK\"}\n", 26) = 26

[pid 12055] <... epoll_wait resumed> {{EPOLLIN, {u32=11, u64=11}}}, 1024, -1) = 1

[pid 12055] recvmsg(11, {msg_name(0)=NULL, msg_iov(1)=[{"{\"cmd\":\"NODE_HANDLE_ACK\"}\n", 65536}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 26

As you can see, the parent and child engage in a protocol through which the socket handle is transported, and the child recreates a net.Socket object from the handle information thus received. Subsequently, the child is able to handle the client underneath this connection.

This protocol, and the way of transporting handles and thereby workload across processes, is the heart and soul of cluster module.

Answer 5

You can't (at least not in Node.js <= 15.13.0) pass a HTTP request or response object to a child process. Hence I came up with a solution using IPC which I will share in this post.

First create a package.json file with this content:

{
  "type": "module"
}

Then create the file server.js with this content:

import * as http from 'http'
import * as child_process from 'child_process'
import * as net from 'net'
import * as fs from 'fs'
import * as crypto from 'crypto'

const ipcPrefix = (process.platform != 'win32' ? '/tmp/' : '\\\\.\\pipe\\') 
                + crypto.randomBytes(8).toString('hex')
const subprocess = child_process.fork('subprocess.js', {
  stdio: ['pipe', 'pipe', 'inherit', 'ipc']
})
let requestNumber = 0

const server = http.createServer()
.listen(8080)
.on('request', (request, response) => {
  if (!subprocess.connected) {
    console.error('Subprocess not connected for: '+request.url)
    response.statusCode = 500
    response.end()
    return
  }
  const {headers, url, method} = request
  const ipcPath = ipcPrefix + requestNumber++ // a unique IPC path
  try {fs.unlinkSync(ipcPath)} catch{} // delete the socket file if it exist already
  let headerChunks = [], headerSize, sizeGotten = 0
  net.createServer() // the IPC server
  .listen(ipcPath, () => subprocess.send({cmd: 'req', headers, url, method, ipcPath}))
  .on('connection', socket => {
    socket.on('close', () => {
      response.end()
      fs.unlinkSync(ipcPath) // clean it up
    })
    socket.once('data', async chunk => {
      headerSize = chunk.readUInt32LE()
      headerChunks.push(chunk)
      sizeGotten += chunk.byteLength
      while (sizeGotten < headerSize) {
        let timer
        const timedOut = await Promise.race([
          // race next packet or timeout timer
          new Promise(resolve => {
            socket.once('data', chunk => {
              headerChunks.push(chunk)
              sizeGotten += chunk.byteLength
              resolve(false)
            })
          }),
          new Promise(resolve => {timer = setTimeout(resolve, 2000, true)})
        ])
        clearTimeout(timer) // to not leave it hanging
        if (timedOut) {
          response.statusCode = 500
          // response.write('lol, bye') // optional error page
          socket.destroy() // this causes the close event which ends the response
          console.error('Subprocess response timed out...')
          return // break the loop and exit function
        }
      }
      const data = Buffer.concat(headerChunks)
      const headerData = data.slice(4, 4+headerSize)
      const header = JSON.parse(headerData.toString())
      response.writeHead(header.statusCode, header.headers)
      if (data.byteLength - 4 - headerSize > 0) {
        response.write(data.slice(4 + headerSize))
      }
      socket.pipe(response) // just pipe it through
      //socket.on('data', response.write.bind(response)) // or do this
      // but set a timeout on the socket to close it if inactive
      socket.setTimeout(2000)
      socket.on('timeout', () => {
        socket.destroy()
        console.log('Subprocess response timed out...')
      })
    })
  })
})

Then create the file subprocess.js . with this content:

import * as net from 'net'

process.on('message', msg => {
  switch (msg.cmd) {
    case 'req': {
      const socket = net.createConnection({path: msg.ipcPath})
      // depending on {headers, url, method} in msg do your thing below
      writeHead(socket, 200, {'testHeader': 'something'})
      socket.end('hello world '+msg.url)
    } break
  }
})

function writeHead(socket, statusCode, headers) {
  const headerBuffer = new TextEncoder().encode(JSON.stringify({statusCode, headers}))
  const headerSize = Buffer.allocUnsafe(4)
  headerSize.writeUInt32LE(headerBuffer.byteLength)
  socket.write(headerSize)
  socket.write(headerBuffer)
}

Now run the server node server and go to localhost:8080 in your browser to see it in action.

If you liked my answer feel free to sponsor me on GitHub.