Problems communicating between Node.js and Python via stdio

Question

I'm spawning a Node process from within a Python thread and passing data between them via stdio. After Python sends something to Node, Node fires up a child process and then sends the output from that child process back to Python.

This works, for a few seconds and then no more data comes. However, if I kill the Node process, then suddenly all of the data comes at once.

I figure this is something to do with buffering but I've tried so many things and can't get it to work properly.

It's worth mentioning that if I run the monitor outside of Python it works fine, so this is probably something on the Python side.

Relevant Python code:

class MonitorThread(Thread):
    # *snip*

    def run(self):
        self.process = Popen(['node',
            path.join(PACKAGE_PATH 'monitor.js')],
            stdout=PIPE, stdin=PIPE, stderr=PIPE)

        while self.process.poll() is None:
            stdout = self.process.stdout.readline().rstrip('\n')

            if stdout:
                main_thread(debug, stdout)

            stderr = self.process.stderr.readline().rstrip('\n')

            if stderr:
                main_thread(debug, stderr)

            #time.sleep(0.1)

Relevant Node.js code (in CoffeeScript but even if you don't know it you get the idea):

# *snip*

child = spawn cmd, options

child.stdout.on 'data', (data) ->
    process.stdout.write "stdout: #{data.toString().trim()}\n"

child.stderr.on 'data', (data) ->
    process.stdout.write "stderr: #{data.toString().trim()}\n"

There's a lot of other code but it's not really relevant, data is being sent and then afterwards data is being received, just only for a moment. It IS still running as when I kill it manually, the rest of the data suddenly appears.

Monitor [send] - {"wrap": false, "directories": {"src": "lib"}, "id": 0, "base_dir": "C:\\Users\\Joe\\Documents\\GitHub\\CoffeeScript-Sublime-Plugin"}
Monitor [recv] - 13:55:30 - compiled src\a.coffee
Monitor [recv] - path.exists is now called `fs.exists`.
Monitor [recv] - 13:55:30 - compiled src\b.coffee

• I've tried util.pump()
• I've tried, with the spawn() call, stdio: 'inherit'
• I've tried waiting for the 'drain' event before sending more data.

Answer 1

You should also pass the end event to close the streams and have node flush them:

child = spawn cmd, options

child.stdout.pipe(process.stdout)
child.stderr.pipe(process.stdout)

child.stdout.on 'data', (data) ->
    process.stdout.write "stdout: #{data.toString().trim()}\n"

child.stdout.on 'end', () ->
    process.stdout.end()

child.stderr.on 'end', () ->
    process.stdout.end()

An even better option would be to use Stream.pipe() .