How to buffer an HTTP response using the request module?

Question

I would like to stream the contents of an HTTP response to a variable. My goal is to get an image via request() , and store it in in MongoDB - but the image is always corrupted.

This is my code:

request('http://google.com/doodle.png', function (error, response, body) {
    image = new Buffer(body, 'binary');
    db.images.insert({ filename: 'google.png', imgData: image}, function (err) {
        // handle errors etc.
    });
})

What is the best way to use Buffer/streams in this case?

Answer 1

The request module buffers the response for you. In the callback, body is a string (or Buffer ).

You only get a stream back from request if you don't provide a callback; request() returns a Stream .

See the docs for more detail and examples.

---

request assumes that the response is text, so it tries to convert the response body into a sring (regardless of the MIME type). This will corrupt binary data. If you want to get the raw bytes, specify a null encoding .

request({url:'http://google.com/doodle.png', encoding:null}, function (error, response, body) {
    db.images.insert({ filename: 'google.png', imgData: body}, function (err) {

        // handle errors etc.

    }); 
});

Answer 2

var options = {
    headers: {
        'Content-Length': contentLength,
        'Content-Type': 'application/octet-stream'
    },
    url: 'http://localhost:3000/lottery/lt',
    body: formData,
    encoding: null, // make response body to Buffer.
    method: 'POST'
};

set encoding to null, return Buffer.

Answer 3

Have you tried piping this?:

request.get('http://google.com/doodle.png').pipe(request.put('{your mongo path}'))

(Though not familiar enough with Mongo to know if it supports direct inserts of binary data like this, I know CouchDB and Riak do.)

Answer 4

Nowadays, you can easily retreive a file in binary with Node 8, RequestJS and async await. I used the following:

const buffer = await request.get(pdf.url, { encoding: null }); 

The response was a Buffer containing the bytes of the pdf. Much cleaner than big option objects and old skool callbacks.