Request does not return complete response. Error JSON.parse

Question

I have a problem in nodejs, I make a request to an api using https.request , the response contains an object of 10000 rows. What happens is that the entire object does not arrive, and parsing gives the error: Unexpected end of JSON input ;

Can someone help?

Function to request:

function request({
  options,
  method,
  resource,
  queryParams,
  bodyParams,
}) {
  return new Promise((resolve, reject) => {
    const hasBodyParams = !!bodyParams;

    const stringifyedQueryParams = strigifyQueryParams(queryParams);

    const optionsRequest = {
      ...options,
      method,
      path: `${resource}${stringifyedQueryParams}`,
    };

    const req = https.request(optionsRequest, (res) => {
      res.setEncoding(configs.ENCODING);
      res.on(events.DATA, data => resolve({
        body: JSON.parse(data),
        statusCode: res.statusCode,
      }));
    });

    req.on(events.ERROR, error => reject(error) );
    hasBodyParams && req.write(bodyParams);
    req.end();
  });
}

Answer 1

As I suspected in the comments, you're not handling multiple data -events.

• When receiving large responses from a request, the data -event is called multiple times, each time with a chunk of data from the response ( not the complete response).
• When you're parsing a chunk, the complete JSON document hasn't been transmitted yet, so the parsing fails with the "Unexpected end of JSON stream" error

In short, you need to:

1. Create a variable to collect the complete body
2. On a data -event, append the new chunk to the complete body
3. When the end -event is called, parse the full body.

Here is a short example, adopted from the official documentation :

https.request(options, (res) => {
  // PARTIAL example
  res.setEncoding("utf8"); // makes sure that "chunk" is a string.
  let fullBody = "";

  res.on("data", data => {
    fullBody += data;
  });
  res.on("end", () => {
    const json = JSON.parse(fullBody);
    // work with json
  });
});