如何流式传输API响应而不是将其全部加载到RAM

Question

I have an API that presents a large data set. Initially, this data set was manageable, but it grows every day and is starting to bog down the server and is causing timeouts, and slow response. Currently, the API loads the entire response into memory before sending it to the client. I'd like to stream the response instead as soon as it is able to.

I followed an online guide on how to construct this API, from here: https://www.freecodecamp.org/news/how-to-build-blazing-fast-rest-apis-with-node-js-mongodb-fastify-and-swagger-114e062db0c9/

I modified it to suit my data, and it worked well enough. However, I've been trying to get it to stream the response now, and I've frankly come up against a wall. I can't seem to adopt the many, many examples online of how to do this into my code.

This is what I am trying to change.

// Get All Metars
exports.getmetars = async (req, reply) => {
  try {
    const flights = await metars.find()
    return flights
  } catch (err) {
    throw boom.boomify(err)
  }
}

This code currently loads the entirety into RAM before sending. How do I stream it instead?

Answer 1

You can stream but you will likely also want to look at enabling pagination, filtering, and/or sorting on your endpoint. Based on the article you linked I'm going to assume you're using Mongoose.

Have a look at this streaming JSON.stringify package . There's a Mongoose example at the bottom of the readme. For you it'd look something like this:

// Get All Metars
exports.getmetars = (req, reply) => {
  try {
    new JsonStreamStringify(metars.find().stream()).pipe(reply)
  } catch (err) {
    throw boom.boomify(err)
  }
}

Referenced answer: https://stackoverflow.com/a/53852868/30697

Also see: Stream from a mongodb cursor to Express response in node.js