无法使用 node-fetch 在 json stream 上解析

Question

I run the sample code from node-fetch stream feature. Sometime it can successfully parse the chunk, sometimes return error msg SyntaxError: Unexpected token { in JSON at position 312

const fetch = require('node-fetch');

async function main() {
  const response = await fetch('https://httpbin.org/stream/3');
  try {
    for await (const chunk of response.body) {
      console.log(JSON.parse(chunk.toString()));
    }
  } catch (err) {
    console.error(err.stack);
  }
}

main()

Anyone know why? Can I rely on the chunk?

Answer 1

By requesting https://httpbin.org/stream/3 , the server sends the data splitting into 3 chunks via stream. The client (in this case your node script) keeps connection with the server and keep receiving the data and splits them into chunks.

The node-fetch simply splits data into chunks every time when a single asynchronous task is completed as you can see here: line 199 of body.js .

So if the splitted data arrives so quickly that the asynchronous task receives multiple chunks of data within a single node's event loop, node-fetch receives multiple jason data.

That's when the error occurs. Run the following code with console.log added. Then you can confirm that when error occurs, the multiple jason objects are kept in a chunk .

const fetch = require('node-fetch');

async function main () {
  const response = await fetch('https://httpbin.org/stream/3');
  try {
    for await (const chunk of response.body) {
      console.log("------chunk-----\n", chunk.toString());
      console.log("Char at 310 -- 315", chunk.toString().substring(310, 315));
      console.log(JSON.parse(chunk.toString()));
    }
  } catch (err) {
    console.error(err.stack);
  }
}

main()

For this site, you can split the data by yourself when the error occurs as follows.

const fetch = require('node-fetch');

async function main () {
  const response = await fetch('https://httpbin.org/stream/3');
  try {
    for await (const chunk of response.body) {
      try {
        console.log(JSON.parse(chunk.toString()));
      } catch (_) {
        console.log("------ Handling multiple chunks ------");
        chunk.toString().split("\n").slice(0, -1).forEach(d => console.log(JSON.parse(d)));
      }
    }
  } catch (err) {
    console.error(err.stack);
  }
}
main()

When you use Fetch API with a browser, you can actually write your own ReadableStreamReader and implement the strategy how to handle the splitted data.

Update:

You can simply use stream-json library's jsonl Parser as follows:

const { parser: jsonlParser } = require('stream-json/jsonl/Parser');

async function main () {
  const response = await fetch('https://httpbin.org/stream/5');
  response.body.pipe(jsonlParser())
    .on('data', ({ key, value }) => console.log(value))
    .on('end', () => console.log("Parsing done."))
    .on('error', err => console.log(err.message));
}

main();