Load Balance: Node.js - Socket.io - Redis

Question

I have 3 Servers running NodeJs, and they are related each other with Redis (1 master, 2 slaves).

The issue i'm having is that running the system on a single server works fine, but when I scale it to 3 NodeJS servers, it starts missing messages and the system gets unstable.

My load balancer does not accept sticky sessions. So every time that the requests from the client arrives to it, they can go to a different server.

I'm pointing all the NodeJS servers to the Redis Master.

It looks like socket.io is storing information on each server and it is not being distributed with redis.

I'm using socket.io V9, I'm suspecting that I don't have any handshake code, could this be the reason?

My code to configure socket.io is:

var express = require('express');
var io = require('socket.io');
var redis = require('socket.io/node_modules/redis');
var RedisStore = require('socket.io/lib/stores/redis');

var pub = redis.createClient("a port", "an ip");
var sub = redis.createClient("a port", "an ip");
var client = redis.createClient("a port", "an ip");
var events = require('./modules/eventHandler');

exports.createServer = function createServer() {
    var app = express();
    var server = app.listen(80);
    var socketIO = io.listen(server);

    socketIO.configure(function () {
        socketIO.set('store', new RedisStore({
            redisPub: pub,
            redisSub: sub,
            redisClient: client
        }));
        socketIO.set('resource', '/chat/socket.io');
        socketIO.set('log level', 0);
        socketIO.set('transports', [, 'htmlfile', 'xhr-polling', 'jsonp-polling']);
    });

    // attach event handlers
    events.attachHandlers(socketIO);

    // return server instance
    return server;
};

Answer 1

Redis only syncs from the master to the slaves. It never syncs from the slaves to the master. So, if you're writing to all 3 of your machines, then the only messages that will wind up synced across all three servers will be the ones hitting the master. This is why it looks like you're missing messages.

More info here .

Read only slave

Since Redis 2.6 slaves support a read-only mode that is enabled by default. This behavior is controlled by the slave-read-only option in the redis.conf file, and can be enabled and disabled at runtime using CONFIG SET.

Read only slaves will reject all the write commands, so that it is not possible to write to a slave because of a mistake. This does not mean that the feature is conceived to expose a slave instance to the internet or more generally to a network where untrusted clients exist, because administrative commands like DEBUG or CONFIG are still enabled. However security of read-only instances can be improved disabling commands in redis.conf using the rename-command directive.

You may wonder why it is possible to revert the default and have slave instances that can be target of write operations. The reason is that while this writes will be discarded if the slave and the master will resynchronize, or if the slave is restarted, often there is ephemeral data that is unimportant that can be stored into slaves. For instance clients may take information about reachability of master in the slave instance to coordinate a fail over strategy.

Answer 2

I arrived to this post:

It can be a good idea to have a "proxy" between nodejs servers and the load balancer. With this approach XHR-Polling can be used in load balancers without Sticky sessions.

Load balancing with node.js using http-proxy

using nodejs-http-proxy i can have custom routing route, ex. by adding a parameter on the "connect url" of socket.io.

Anyone tried this solution before?