在Node.js中使用流来缓冲HTTP通信

Question

I'm trying to use streams in Node.js to basically build a running buffer of HTTP data until some processing is done, but I'm struggling with the specifics of streams. Some pseudocode will probably help:

var server = http.createServer(function(request, response) {

    // Create a buffer stream to hold data generated by the asynchronous process
    // to be piped to the response after headers and other obvious response data
    var buffer = new http.ServerResponse();

    // Start the computation of the full response as soon as possible, passing
    // in the buffer stream to hold returned data until headers are written
    beginAsyncProcess(request, buffer);

    // Send headers and other static data while waiting for the full response
    // to be generated by 'beginAsyncProcess'
    sendObviousData(response, function() {

        // Once obvious data is written (unfortunately HTTP and Node.js have
        // certain requirements for the order data must be written in) then pipe
        // the stream with the data from 'beginAsyncProcess' into the response
        buffer.pipe(response);
    });
});

Most of this is almost legitimate code, but it doesn't work. The basic issue is figuring out a way to take advantage of the asynchronous nature of Node.js when there are certain order requirements associated with HTTP requests, namely that headers must always be written first.

While I would definitely appreciate any answers with little hacks to get around the order problem without directly addressing streams, I wanted to use the opportunity to get to know them better. There are plenty of similar situations, but this scenario is more to open the can of worms than anything else.

Answer 1

Let's make a use of callbacks and streams in Node.js and .pause() / .resume() stream functions:

var server = http.createServer(function(request, response) {

    // Handle the request first, then..

    var body = new Stream(); // <-- you can implement stream.Duplex for read / write operations
        body.on('open', function(){
            body.pause();
            // API generate data
            // body.write( generated data ) <-- write to the stream
            body.resume();
        });

    var firstPartOfThePage = getHTMLSomeHow();

    response.writeHead(200, { 'Content-Type': 'text/html'});

    response.write(firstPartOfThePage, function(){ // <-- callback after sending first part, our body already being processed
        body.pipe( response ); // <-- This should fire after being resumed
        body.on('end', function(){
            response.end(); // <-- end the response
        });
    });
});

Check this: http://codewinds.com/blog/2013-08-31-nodejs-duplex-streams.html for costum duplex stream creation.

Note: it's still a pseudo code