Node.js HTTPS.get returns a status code 406

Question

I'm trying to make a promise-ified HPPTS request to an API, using only the built in HTTPS module. The returned status code is 406, and I'm also getting a buffer error:

TypeError: buf.copy is not a function
at Function.Buffer.concat (buffer.js:240:9)
at IncomingMessage.<anonymous> (/var/task/index.js:562:41)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:905:12)
at nextTickCallbackWith2Args (node.js:437:9)
at process._tickDomainCallback (node.js:392:17)

Here's my function:

function createRequest(url, body, callback){
  return new Promise(function(resolve, reject) {
    try{
        var parsedUrl = Url.parse(url, true, true);
    }
    catch(e) {
        console.log("URL parsing error");
    }

    try{
    https.get({
        hostname: parsedUrl.hostname,
        path: parsedUrl.path,
        headers: {
            'Content-Type': 'application/json'
        }
    }, function(response) {
    console.log(response.statusCode);
    response.setEncoding("utf8");
    var responseBuffer = [];
    response.on('data', function(d) {
        responseBuffer.push(d);
    });

    response.on('end', function() {
        var responseString = Buffer.concat(responseBuffer);
        callback(JSON.parse(responseString));
        resolve(responseString);
    });

    response.on('error', (e) => {
        reject(e);
    });
  });
  } catch(e){
    console.log(e);
  }
  });
}

At the response end, the responseText is just a single space.

So what am I doing wrong here? Thank you for the help and patience.

EDIT: Also worth noting, if I change the Buffer.concat line to var responseString = responseBuffer.join(); the error becomes this, on the callback(JSON.parse(responseString)); line.

SyntaxError: Unexpected end of input
at Object.parse (native)
at IncomingMessage.<anonymous> (/var/task/index.js:564:27)
at emitNone (events.js:72:20)
at IncomingMessage.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:905:12)
at nextTickCallbackWith2Args (node.js:437:9)
at process._tickDomainCallback (node.js:392:17)

Answer 1

It looks like there are 2 separate problems:

• When you call response.setEncoding('utf8') , node automatically converts the incoming data into strings. This means that the data event fires with strings, not Buffer s.

  This means you either need to keep the stream in binary mode (by not calling setEncoding ), keep the array of buffers, and then concatenate and convert them to a string at the end.

   response.on('end', function() { try { var responseString = Buffer.concat(responseBuffer).toString('utf8'); resolve(JSON.parse(responseString)); } catch(ex) { reject(ex); } });

  ...or keep the setEncoding call and do simple string concatenation.

   response.on('data', function(str) { responseString += str; });

  I recommend the former for better performance (node has to do some internal buffering to deal with multibyte characters in streaming mode).

• The API you're using is returning a 406 ( Not Acceptable ). This likely means you must supply an Accept header in your request.

Answer 2

Based on the following doc, Buffer.concat needs List of Buffer or Uint8Array as parameter

https://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength

You can try something like the following to convert the string to buffer before push it to your array.

response.on('data', function(d) {
    responseBuffer.push(Buffer.from(d, 'utf8'));
});

Check the detailed issue here

https://github.com/nodejs/node/issues/4949

Hope it helps