Node.js client request hangs

Question

I have a node.js process that uses a large number of client requests to pull information from a website. I am using the request package ( https://www.npmjs.com/package/request ) since, as it says: "It supports HTTPS and follows redirects by default."

My problem is that after a certain period of time, the requests begin to hang. I haven't been able to determine if this is because the server is returning an infinite data stream, or if something else is going on. I've set the timeout, but after some number of successful requests, some of them eventually get stuck and never complete.

var options = { url: 'some url', timeout: 60000 };
request(options, function (err, response, body) {
    // process
});

My questions are, can I shut down a connection after a certain amount of data is received using this library, and can I stop the request from hanging? Do I need to use the http/https libraries and handle the redirects and protocol switching myself in order the get the kind of control I need? If I do, is there a standardized practice for that?

Edit: Also, if I stop the process and restart it, they pick right back up and start working, so I don't think it is related to the server or the machine the code is running on.

Answer 1

Note that in request(options, callback) , the callback will be fired when request is completed and there is no way to break the request.

You should listen on data event instead:

var request = require('request')

var stream = request(options);

var len = 0
stream.on('data', function(data) {
   // TODO process your data here

   // break stream if len > 1000
   len += Buffer.byteLength(data)
   if (len > 1000) {
      stream.abort()
   }
})