如何限制来自 fetch API 的流以节省带宽？

Question

How do I get the fetch API to cut off the stream after a few hundred bytes? I only need the meta description from the top of the page, and the rest of the page is upwards of 500k

Answer 1

Check if the website has a feature that allows you to send a flag that tells them to only send the meta description.

If it doesn't, then your machine is going to receive the data no matter what. If you want to save time reading the whole response, you can just read until you have all the meta information, then stop reading and discard the rest. This will just save you some I/O time, not bandwidth.

Answer 2

You can use the Response.body stream to read the received data bit by bit and use an AbortSignal to cancel the request when you have received enough data.

Note that you cannot control how much data you will receive in each chunk, so you might receive more than needed, but I would assume that with 500 KB, the data will arrive in multiple chunks, so by canceling you will save some data.

Here is some example code that will abort as soon as it has received at least 10 KB:

const LIMIT = 10000;
const abort = new AbortController();
const res = await fetch(url, {
    signal: abort.signal
});

let bodySoFar = '';
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
while (true) { // eslint-disable-line no-constant-condition
    const { done, value } = await reader.read();
    if (done) {
        break;
    }
    bodySoFar += value;
    if (bodySoFar.length > LIMIT) {
        abort.abort();
        break;
    }
}

Note that TextDecoderStream does not have good browser support yet. As a workaround, you can collect the data in an Uint8Array instead.