How can I fix a memory problem in node.js?

Question

I get the following error. As I checked, it seems that the json file is too large to run because of insufficient server memory.

How can we solve this problem? There is a thing called Node streaming, it's hard for me to understand. Can you explain it more easily?

server.js

const express = require('express');
const cors = require('cors');
const userJson = require('./user');
const locationJson = require('./location');

const API_PORT = process.env.PORT || 3002;
const app = express();
app.use(cors());
const router = express.Router();

router.get("/getUserData", (req, res) => {
    return res.json(userJson);
});

router.get("/getLocationData", (req, res) => {
    return res.json(locationJson);
});

app.use("/api", router);

app.listen(API_PORT, () => console.log(`LISTENING ON PORT ${API_PORT}`));

error messages

buffer.js:585
      if (encoding === 'utf8') return buf.utf8Slice(start, end);
                                          ^

Error: Cannot create a string longer than 0x3fffffe7 characters
    at stringSlice (buffer.js:585:43)
    at Buffer.toString (buffer.js:655:10)
    at Object.readFileSync (fs.js:392:41)
    at Object.Module._extensions..json (internal/modules/cjs/loader.js:816:22)
    at Module.load (internal/modules/cjs/loader.js:666:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:606:12)
    at Function.Module._load (internal/modules/cjs/loader.js:598:3)
    at Module.require (internal/modules/cjs/loader.js:705:19)
    at require (internal/modules/cjs/helpers.js:14:16)
    at Object.<anonymous> (/Users/k/Desktop/dev/backend/location.js:1:22)

Answer 1

You can stream the file and pipe() it directly into the res:

const fs = require('fs');
const path = require('path');

const getBigJson = (req, res, next) => {
    const pathToJson = path.join(__dirname, 'path/to/file');
    const jsonStream = fs.createReadStream(pathToJson);
    res.set({'Content-Type': 'application/json'});
    jsonStream.pipe(res);
};

It should send it chunk by chunk without exhausting your memory.

See https://github.com/substack/stream-handbook for a good explanation of streams.