How to use fs and JSONStream to write a huge file without trigger RangeError: Maximum call stack size exceeded

Question

What I did here is trying to replace JSON.stringify() with something else, to avoid RangeError. And I found the method below.

This is my code: (I use big-json module and this module implement json-stream-stringify for stringify)

const data = generateData();       
                   
const stringifyStream = json.createStringifyStream({  
    body: data
});

stringifyStream.on('data', function(str) {
    console.log(str.toString());
})

I tried to print the str , it will print the string type data line by line, my question is: how to write the complete str to a file? I tried to add fs.writefileSync() after the console.log() and it will only write in the last line of str . Sorry I am new to stream and node.js. Hvae no idea how to fix it.

The other question is, can I find a way to re-format the output string just like JSON.stringify(data, null, 2) did?

FYI: The data is a really deep object. No circular, but may have repeated part. And I prefer to keep everything unchanged and not pruned.

Really appreciate any help here. Thank you for your time.

Answer 1

Have you tried separating JSON.stringify from fs.writeFileSync? just to know wich one is actually having problems.

If it is JSON.stringify I suggest you to fragment the object in multiple ones or filter what you use. Whatever you can do to reduce the actual data. We don't know what de data really is so I cannot make assumptions on that but you never have those problems generally, if it's caused by the data being too big, you are doing things wrong!

If the problem is fs.writeFileSync and you have your JSON stringified then try using fs.createWriteStream to write the data.