Is there a reason using setTimeout within an on-data event from a child process causes arbitrarily long delays?

Question

The Problem:

I have encountered a scenario using Javascript in NW.js whereupon setTimeout() will experience arbitrarily long delays instead of executing the callback after the requested duration. Generally anywhere from a few dozen milliseconds to thousands of milliseconds of unexpected delay.

Of course, setTimeout() doesn't guarantee it will execute the callback precisely when requested, but it's usually very, very close.My expectation would be that this would always be the case, so long as the main thread wasn't overloaded with other tasks. For instance, if I mash F12 and bring up my browser console and run the below:

var ts = Date.now();
setTimeout(function () {
  console.log(`This should run immediately! It took ${Date.now() - ts}ms`);
}, 0);

I would expect output like:

This should run immediately! It took 1ms This should run immediately! It took 1ms (possibly 0-5ms, depending - but definitely very little time, and repeatably so if you run it again and again)

However, in the aforementioned scenario, the delay can be significant; often thousands of milliseconds longer than the duration requested.

The Scenario:

All it seemingly requires is for setTimeout() to be used after an onData event from a child process. It can be used directly in the process.stdout.on('data') callback, or it can be used some time later in the chain of execution that might follow. In both cases, setTimeout() seems to misbehave.

Nothing else misbehaves; Javascript continues to execute as normal, Promises resolve in a timely fashion, etc etc... just so long as setTimeout() isn't involved. Even horrible things like using For loops to impose a delay instead of setTimeout() will work reliably.

I have prepared an example which replicates what I am observing on my systems, in the hopes others could test it and see if perhaps I am doing something wrong!

I am hoping someone will turn around and tell me I've overlooked something!

Please see below for the code. I've tried it on multiple machines with the same behaviour observed. Please note the test case given is the bare minimum example I've thus far discovered which replicates the issue. It is not representative of what I am actually trying to do in practice ; instead, it just attempts to replicate the misbehaving setTimeout() issue I have encountered on my project (and it does replicate it, as far as I can tell).

Working Example:

This scenario requires:

• NW.js: latest 0.49.2 SDK, also 0.48.3 SDK tested (same behaviour)
• Python: latest 3.9.0, also 3.8.5 tested (same behaviour)
• Windows: v2004 builds (19041.508) and (19041.572) were tested (same behaviour)

Download and unzip the relevant NW.js SDK, then plonk the following files in the folder and run nw.exe . You should see a white window appear; open devtools (F12) and perform a test with td.beginBasicTest() :

package.json:

{
  "name": "STT",
  "description": "setTimeout Test",
  "main": "test.html",
  "node-remote": "http://127.0.0.1",
  "window": {
    "title": "test",    
    "show": true,
    "toolbar": true,
    "fullscreen": false,
    "width": 500,
    "height": 500,
    "frame": false,
    "position": "center",
    "resizeable": true
  },
  "dependencies": {    
  },
  "devDependencies": {},
  "chromium-args": "--password-store=basic --disable-pinch --disable-background-timer-throttling --disable-raf-throttling"
}

child_process.py:

#!/usr/bin/env python3
import sys
import asyncio
import json

class TestProcess:
    def __init__(self):
        self.alive = True
        self.run = True        
            
    def __hello(self):
        hello_msg = '{"resp": "hello"}'
        print(hello_msg, flush=True)
            
    async def __stdin_msg_handler(self, inp):
        # read the input command and respond accordingly
        if(len(inp) > 0):
            try:
                # convert from JSON to dict
                msg = json.loads(inp[0])
            except:
                # couldn't decode message; ignore it?
                print('{"resp":"badCommand"}', flush=True)
                msg = None
            if(msg):
                if(msg["cmd"] == "hello"):
                    self.__hello()

    async def __listen(self):
        line = await self.loop.run_in_executor(None, sys.stdin.readline)
        if(len(line) > 0):
            msgs = line.split("\r")
            return msgs
        else:            
            return False

    async def __loop(self):
        while self.run == True:
            msgs = await self.__listen()
            if(msgs):
                await self.__stdin_msg_handler(msgs)

    def start(self):
        self.loop = asyncio.get_event_loop()
        self.loop.run_until_complete(self.__loop())


if __name__ == '__main__':
    tproc = TestProcess()
    tproc.start()

test.html:

<head>

<script>
const spawn = require(`child_process`).spawn; // ability to execute background tasks/commands

class TestDelay {
  constructor (delay = 5) {
    this.childProcessPath = `child_process.py`;
    this._startProcess();
  }

  beginBasicTest (delay = this.defaultDelay) {
    this._pTestProcedureWithComms(delay)
      .then(() => {
        console.log(`Test completed - did it work? Try it a few times, performance seems to vary...`);
      });
  }

  _startProcess () {
    this.childProcess = spawn(`py`, [`-3`, this.childProcessPath], { stdio: `pipe` });
    this.childRunning = true;

    this.childProcess.stdout.on(`data`, (d) => {
      console.log(`stdout: ${d}`);
      var ts = Date.now();
      setTimeout(function () {
        console.log(`This should run immediately! It took ${Date.now() - ts}ms`);
      }, 0);
      setTimeout(function () {
        console.log(`This should run after 5s! It took ${Date.now() - ts}ms`);
      }, 5000);
    });

    this.childProcess.stderr.on(`data`, (d) => {
      console.warn(`stderr: ${d}`);
    });

    this.childProcess.on(`exit`, (code) => {
      this.childRunning = false;
      console.log(`child process exited with code ${code}`);
    });

    this.childProcess.on(`close`, (code) => {
      console.log(`child process IO closed with code ${code}`);
    });
    console.log(`Started child process...`);
  }

  _pTestProcedureWithComms (delay = this.defaultDelay) {
    return new Promise(resolve => {
      this._pSendToProcess({ cmd: `hello` })
        .then(resolve);
    });
  }

  _pSendToProcess (text = ``) {
    return new Promise(resolve => {
      this.childProcess.stdin.write(JSON.stringify(text), `utf-8`);
      this.childProcess.stdin.write(`\n`, `utf-8`);
      resolve();
    });
  }
}

var td = new TestDelay();
</script>
</head>
<body>
Hello!
</body>

When I run the tests a few times, I frequently observe setTimeout seeming to misbehave. Here is some example output I got while running the test a few times:

Answer 1

the fact that sometimes your immediate/5 second reports at 1ms/5010ms and sometimes reports as 8714ms/8715ms tells mes that this is an issue where your event loop is full and waiting on something else to complete before the setTimeout can run.

I don't know enough about stdio/stdout/stderr to offer much more than that.