Node.js https.request with keep-alive header

Question

I want to pull posts from a users Facebook wall.

The following code snippet works, but it never terminates:

var https = require('https');

facebookWall = function(user) {
  var options = {
    host: 'graph.facebook.com',
    port: 443,
    path: '/me/home?access_token=' + user.facebook_token + '&since=' + encodeURIComponent(user.facebook_timestamp),
    method: 'GET',
    headers: {
      'Connection':'keep-alive'
    }
  };

  var req = https.request(options)
  .on('response', function(response) {
    var body = '';
    response.on('data', function(data) {
      body += data;
      try {
        var wallPosts = JSON.parse(body);
        console.log("user " + user.id + " has " + wallPosts.data.length + " new items on their wall");
      }
      catch (e) {
        //console.log("waiting for more data chunks...");
      }
    })
  });
  req.end();

  req.on('error', function(e) {
    console.error(e);
  });
}

I think it is caused by the 'Connection':'keep-alive' header. When I replace it with 'Connection':'close' the script will terminate when all data has been retrieved from facebook.

I'm hoping to be able to use the keep-alive header to prevent having to create a new SSL connection for each request. I have thousands of requests and with the keep-alive header, it completes in just a few seconds, as opposed to a few minutes without the keep-alive header.

Does anyone know how to accomplish this? I'm fairly new to Node.JS, so if I'm missing something obvious, I apologize.

Answer 1

It's because keep-alive is not yet implemented for https/tls/ssl in node 4.x and I believe for 6.x too. That's why in node websocket-server it doesn't work as well, see https://github.com/nephics/node-websocket-server/commit/3a732bff6aabe694834d87086a7718be7c0ce138

Answer 2

I notice you're using https. Depends on what version of node you're using, but there is a known issue with the end event right now.

https://github.com/joyent/node/issues/728

Answer 3

You have to do the following:

1) Put response.on('end', function() {... do the output... }); for the https.request Don't output data in the response.on('data', ...);

2) Use Connection:keep-alive, "close" will cause very poor performance issue. I have done lot of testing and I can confirm this.

Other than that:

3) In your options, set the agent, and set agent.maxSockets to a larger number if you need concurrency. default is only 5.

4) You should consider to make your own routine to handle https.request timeout. (please go to github/joyent and search for it. basically use setTimeout to emit a timeout error).