存储为 JavaScript 缓冲区的 Un-TAR 和 un-GZip 文件

Question

I am developing a server script on Node.js/Express.js that receives uploaded .tar.gz archives with multiple files. The script has to untar and ungzip CSV files in archives, parse them and store some in database. There is no need to store files on the server, just process them. To upload files I am using Multer without specifying where to store files, so file uploads are only available in req.files as Buffer .

My question is, how is it possible to untar and ungzip Buffer to get the contents of the files? If I do something like:

const { unzipSync } = require('zlib');

const zipped = req.files[0];
const result = await unzipSync(zipped.buffer);
const str = result.toString('utf-8');

I get not the content of the file, but all information including file name, some metadata etc as string, which is tricky to parse. Is there a better way?

Answer 1

I managed to untar and unzip Buffer using tar-stream and streamifier libraries.

const tar = require('tar-stream');
const streamifier = require('streamifier');
const { unzipSync } = require('zlib');

const untar = ({ buffer }) => new Promise((resolve, reject) => {
  // Buffer is representation of .tar.gz file uploaded to Express.js server
  // using Multer middleware with MemoryStorage
  const textData = [];
  const extract = tar.extract();
  // Extract method accepts each tarred file as entry, separating header and stream of contents:
  extract.on('entry', (header, stream, next) => {
    const chunks = [];
    stream.on('data', (chunk) => {
      chunks.push(chunk);
    });
    stream.on('error', (err) => {
      reject(err);
    });
    stream.on('end', () => {
      // We concatenate chunks of the stream into string and push it to array, which holds contents of each file in .tar.gz:
      const text = Buffer.concat(chunks).toString('utf8');
      textData.push(text);
      next();
    });
    stream.resume();
  });
  extract.on('finish', () => {
    // We return array of tarred files's contents:
    resolve(textData);
  });
  // We unzip buffer and convert it to Readable Stream and then pass to tar-stream's extract method:
  streamifier.createReadStream(unzipSync(buffer)).pipe(extract);
});

Using this approach I managed to avoid storing any temporary files on filesystem and process all files' contents in memory exclusively.