node.js calculating bandwidth usage by domain

Question

How can I monitor the bandwidth usage of each domain using node.js as the web server?

Does anyone know of an API call I haven't come across to do this?

Or a module or another method which others have used in a multi-tenant environment where you are charging by bandwidth?

Update:

Does anyone know of a lightweight proxy / server which could be put in front of any web server (node.js, apache, etc) which can record these bandwidth stats by inspecting the domain?

Answer 1

Without modifying the node.js core, the best option seems to be to track it at the socket level using the bytesRead and bytesWritten variables.

This is actually more accurate than merely measuring the size of http request / response string as some other bandwidth trackers do to calculate data transferred.

You have 2 options: 1) log the bytes at each request OR 2) log the bytes on the TCP connection close.

Logging on the request level will provide real-time data which could be useful. However, it is likely to slow things down depending on how you implement the logging. It is probably best to keep it in memory and dump it to disk / DB every so often.

Log at request level:

var http = require('http');
var server = http.createServer(function (req, res) {
  // setup a counter for bytes already logged
  req.connection.bytesLogged = (req.connection.bytesLogged || 0);

  // output the bytes read by the socket
  console.log('Socket [' + req.connection.remoteAddress + '] [' + req.headers.host + '] - Bytes Read: ' + req.connection.bytesRead);

  // calculate the bytes of the request (includes headers and other tcp overhead - more realistic)
  req.bytes = req.connection.bytesRead - req.connection.bytesLogged;
  // output the bytes size of the request (note this is calculated after the TCP packets have been collected from the network and parsed by the HTTP parser
  console.log('Request [' + req.connection.remoteAddress + '] [' + req.headers.host + '] - Bytes: ' + req.bytes);

  // log the bytes to a memory array, DB or disk (implementation not shown)
  // [code here]

  // add the request bytes to the counter
  req.connection.bytesLogged = req.connection.bytesLogged + req.bytes;

  // normal http server processing like return document or file
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('ok');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.1.1:1337/');

Log at socket level:

var http = require('http');
var server = http.createServer(function (req, res) {

  // create some variables for data we want to log
  //
  // due to the way node.js works the remoteAddress and remotePort will not
  // be available in the connection close event
  // we also need to store the domain/host from the http header because the http
  // request also won't exist when the connection close event runs
  var remoteAddress = req.connection.remoteAddress;
  var remotePort = req.connection.remotePort;
  var host = req.headers.host;
  var connection = req.connection;

  // output bytes read by socket on each request
  console.log('HTTP Request [' + remoteAddress + ':' + remotePort + '] [' + host + '] - connection Bytes Read: ' + connection.bytesRead);

  // setup handle for connection close event
  //
  // to avoid the handle being added multiple times we add the handle onto
  // the connection object which will persist between http requests
  //
  // we store the handler so we can check whether it has already been added
  // to the connection listeners array - a less robust alternative would be to
  // add a flag like connection.closeHandleAdded = true
  connection.handle = connection.handle || {};
  if (!connection.handle.onconnectionClose) {
    connection.handle.onconnectionClose = function() {
      onconnectionClose(remoteAddress, remotePort, host, connection.bytesRead);
    }
  }

  // check whether the close handle has already been added to the connection
  // if not add it
  if(connection.listeners('close').indexOf(connection.handle.onconnectionClose) == -1)
  {
    // attach handler to connection close event
    connection.on('close', connection.handle.onconnectionClose);
    // set connection idle timeout to 5 secs for testing purposes (default is 2min)
    connection._idleTimeout = 5000;
  }

  // process http request as required
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('ok');

}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.1.1:1337/');

function onconnectionClose (remoteAddress, remotePort, host, bytesRead) {
   console.log('connection Closed [' + remoteAddress + ':' + remotePort + '] [' + host + '] - connection Bytes Read: ' + bytesRead);
}

Answer 2

You should have a look at Http Trace .

It is a node.js module that captures, decodes and analyses HTTP and WebSocket traffic. It can find the size and domain for each request, so with some tweaking, you should be able to do what you want to accomplish here.

It is working well on my Ubuntu server with node v0.6.15, all I had to do was to "apt-get install libpcap0.8-dev".

Answer 3

Try to use this module to measure HTTP response/request: https://github.com/hex7c0/transfer-rate

Data is sent between a client and a server by a socket. Each HTTP response/request has their own socket. The module gets sent-byte in the socket ( socket.bytesWritten for the response, and socket.bytesRead for the request). The time for sending the data is calculated by a command ( process.hrtime(start) ). Then, we divide the results to get the actual transmitted rate.