节点上传的图像保存 - 流与缓冲区

Question

I am working on image upload and don't know how to properly deal with storing the received file. It would be nice to analyze the file first if it is really an image or someone just changed the extension. Luckily I use package sharp which has exactly such a feature. I currently work with two approaches.

Buffering approach

I can parse multipart form as a buffer and easily decide whether save a file or not.

const metadata = await sharp(buffer).metadata();
if (metadata) {
  saveImage(buffer);
} else {
  throw new Error('It is not an image');
}

Streaming approach

I can parse multipart form as a readable stream. First I need to forward the readable stream to writable and store file to disk. Afterward, I need again to create a readable stream from saved file and verify whether it is really image. Otherwise, revert all.

// save uploaded file to file system with stream
readableStream.pipe(createWriteStream('./uploaded-file.jpg'));

// verify whether it is an image
createReadStream('./uploaded-file.jpg').pipe(
  sharp().metadata((err, metadata) => {
    if (!metadata) {
      revertAll();
      throw new Error('It is not an image');
    }    
  })
)

It was my intention to avoid using buffer because as I know it needs to store the whole file in RAM. But on the other hand, the approach using streams seems to be really clunky.

Can someone help me to understand how these two approaches differ in terms of performance and used resources? Or is there some better approach to how to deal with such a situation?

Answer 1

In buffer mode, all the data coming from a resource is collected into a buffer , think of it as a data pool, until the operation is completed; it is then passed back to the caller as one single blob of data. Buffers in V8 are limited in size. You cannot allocate more than a few gigabytes of data, so you may hit a wall way before running out of physical memory if you need to read a big file.

On the other hand, streams allow us to process the data as soon as it arrives from the resource. So streams execute their data without storing it all in memory. Streams can be more efficient in terms of both space (memory usage) and time (computation clock time).