Unexpected end of JSON input after HTTP request

Question

Here I'm making a HTTPS request and storing the JSON sent back in the data variable. I have used a similar function elsewhere in the program where it worked as expected. However this time I'm getting a SyntaxError: Unexpected end of JSON input on the line where is use JSON.parse(data) .

I suspect data doesn't contain all of the JSON sent back from the API since it is quite large. Do I need to change how I capture the input from the HTTPS request incase it is not all captured into data in one go or is there some other issue here?

The only thing making my question the code instead of the API is that when making the request on a third party HTTP client I get all the JSON expected.

function isStreamLive(channelID, callback) {
    isLive = false;
    const optionsStreamInfo = {
        'hostname': 'api.twitch.tv',
        'path': '/kraken/streams/' + channelID,
        'headers': {
            'Client-ID': 'xxxxxxxxxxxxxxxxxxxxxxxxxx',
            'Accept': 'application/vnd.twitchtv.v5+json'
        }
    }
    const request = https.get(optionsStreamInfo, (result) => {
        result.on('data', (data) => {
            console.log(JSON.parse(data));
        });
    });
    request.end();
}

The output before the program terminates shows the vast majority of the JSON expect but cuts off and is followed by a few blank lines.

Logging the data variable without using JSON.parse shows

<Buffer 7b 22 73 74 72 65 61 6d 22 3a 7b 22 5f 69 64 22 3a 33 38 31 34 30 30 37 33 38 30 38 2c 22 67 61 6d 65 22 3a 22 44 6f 74 61 20 32 22 2c 22 62 72 6f 61 ... 
1321 more bytes>
<Buffer 76 69 65 77 73 22 3a 34 38 37 38 39 31 32 2c 22 66 6f 6c 6c 6f 77 65 72 73 22 3a 39 31 30 33 39 2c 22 62 72 6f 61 64 63 61 73 74 65 72 5f 74 79 70 65 ... 
84 more bytes>

From this I assume that JSON.parse is taking in the first Buffer and not the second leading to the JSON being cut off.

Is there a way to get take in these two buffers at the same time?

Answer 1

I don't believe you need to end the request.

You can do something like this:

function isStreamLive(channelID, callback) {
    isLive = false;
    const optionsStreamInfo = {
        'hostname': 'api.twitch.tv',
        'path': '/kraken/streams/' + channelID,
        'headers': {
            'Client-ID': 'xxxxxxxxxxxxxxxxxxxxxxxxxx',
            'Accept': 'application/vnd.twitchtv.v5+json'
        }
    }
    const request = https.get(optionsStreamInfo, (result) => {
        result.on('data', (data) => {
            const mergedData = Buffer.concat(data)
            console.log(JSON.parse(mergedData));
        });
    });
}

Using Request-promise, you can also use node-fetch, its very similar.. const rp = require('requrest-promise');

const optionsStreamInfo = {
    uri: `https://api.twitch.tv/kraken/streams/${channelID}`,
    method: 'GET',
    headers: {
        'Client-ID': 'xxxxxxxxxxxxxxxxxxxxxxxxxx',
        'Accept': 'application/vnd.twitchtv.v5+json'
    },
    json:true // this should take care of this buffer business
}

return rp(optionsStreamInfo).then((res)=>{
    console.log(res)
}).catch((err)=>{
    console.log(err)
})