Piping oboe.js parsed json to http response object in Node.js

Question

I'm using Oboe.js to parse JSON from a Node readStream and I want to send this back to the client requesting it in a memory-efficient manner. Is it possible to pipe data from Oboe's node or path events to a Node.js HTTP response object, so I can delivered parsed data on the fly to a client rather than collecting it in full and sending it all at once? This is the code I have so far:

const express = require('express');
const app = express();
const PORT = 3000;
const oboe = require('oboe');
const fs = require('fs');

app.get('/download', (req, res) => {

    const jsonDataStream = fs.createReadStream('./citylots.json');

    oboe(jsonDataStream)
        .node('features.*', function(feature) {
           // res.send is non-streaming. how to stream?
            res.send(feature);
        });
});

app.listen(PORT, () => console.log(`up on port ${ PORT }!`));

Answer 1

For your specific question, I think the problem has to do with the fact that res.send terminates the response ( docs ). Given that res is a writable stream, you can call res.write to write a string to the response. You will have to do some additional work to insert commas as well as a starting [ and ] .

const express = require('express');
const app = express();
const PORT = 3000;
const oboe = require('oboe');
const fs = require('fs');

app.get('/download', (req, res) => {

    const jsonDataStream = fs.createReadStream('./citylots.json');

    oboe(jsonDataStream)
        .node('features.*', function(feature) {
          res.write(JSON.stringify(feature));
        });
});

app.listen(PORT, () => console.log(`up on port ${ PORT }!`));

To step back from this though, I want to be sure Oboe is good for your use case. It might be complicating your implementation without much benefit. Is your goal to reduce memory by sending less data to the client? In other words, are you trying to avoid sending all of citylots.json and instead want to only send its features ?

Answer 2

I think what you need is keeping sending data without holding the entire file in memory.

Therefore, you need to use pipe. Pipe will allow you to send the data with chunks