request: how to pipe file and keep the response object

Question

I use npm request. I want to download and write a file to the filesystem and after that-use the returned response object for further processing.

I know that I can pipe the response object directly but afterwards I have no response object more for further processing

var result = await request(builtReq).pipe(fs.createWriteStream(myFilePath));

So my implementation at the moment looks like this:

var result = await request(builtReq);

But I am not able to pipe the result object because the streamable state is false.

So I need a way to keep the return of the request and to write the file to the filesystem. Can I reset the stream state or somehow keep the response obj and write the file at the same time? I tried to implement the write file manually via. fs.writeFile() after I received the response obj but I had problems with file encodings because I can receive anything and I ended up with broken files.

Has somebody an idea how to solve that?

Answer 1

Do you want the response object (this contains the status code and headers), or do you need the response body in-memory?

If you only want the response object, and still want the body written to a file, you can grab the response like so:

var response;

await request(builtReq).on('response', r => {
    console.log('Response received: ' + r);
    response = r;
}).pipe(fs.createWriteStream(myFilePath));

console.log('Finished saving file, response is still ' + response);

If you need to process the actual body, you have a couple options:

• Keep your existing code as-is, then just read the file off the disk right after it finishes (since you are using await , that would be the next line) and process it in memory.

• Pipe the input stream to multiple places -- one to the file write stream, and the other to an in-memory buffer (using a standard on('data') handler). Once the stream finishes, the file is saved and you can process the in-memory buffer. See related question node.js piping the same readable stream for several different examples.

Answer 2

remember promises can only be resolved once.

so to solve your problem:

var promise = request(builtReq)

// now do what ever number of operations you need
result.pipe(fs.createWriteStream(myFilePath));

promise.then(r=>console.log('i will still get data here :)');
promise.then(r=>console.log('and here !');
promise.then(r=>console.log('and here !!');

var result = await promise; // and here to !!

a promise once resolve, the .then will always get the result.