Understanding how to use Redis with Node.js and Server Sent Events

Question

My Project is built with Nodejs as proxy server to communicate with an external API.

The API send product updates via Redis (pub/sub); The Proxy server handle the message and send it to the client via SSE (Server Sent Events).

It is the first time for me using Redis and SSE and looking online for tutorials seems to be easy to implement and I did it.

On the Client side I just created an EventSource and as soon as I receive an update I do something with it:

// Client Side
var source = new EventSource('/redis'); // /redis is path to proxy server
source.addEventListener('items', handleItemsCallback, false);
source.addEventListener('users', handleUsersCallback, false);
source.addEventListener('customers', handleCustomersCallback, false);

// Function sample...
function handleItemsCallback (msg) {
   // Do something with msg...
}

In the Proxy server I created a controller with routing to /redis to handle Redis messages:

exports.redisUpdates = function (req, res) {
    // Redis Authentication
    var redisURL = url.parse(process.env.REDISCLOUD_URL);
    var client = redis.createClient(redisURL.port, redisURL.hostname, {ignore_subscribe_messages: false});
    client.auth(redisURL.auth.split(":")[1]);

    // let request last as long as possible
    req.socket.setTimeout(0);

    // Subscribe to channels
    client.subscribe('items', 'users', 'customers');

    // Handle messages
    client.on('message', function (channel, message) {
        res.write('retry: 5000\n');
        res.write('event: ' + channel + '\n');
        res.write('data: ' + message + '\n\n');
        res.flush(); // If I do not add this it doesn't push updates to the client (?)
    });

    //send headers for event-stream connection
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive'
    });
    res.write('\n');

};

Using it locally in a development environment it works fine but using it in Production generate several different errors, the App is hosted on Heroku and the Heroku Metrics show several H18 , H12 , H27 Errors;

Sometimes the /redis call return status 503 ;

What I wish to understand is if I'm using those services correctly, why all tutorials do not mention res.flush() and I discovered it by myself to let it work the first time...

Answer 1

In all fairness, this question is not really answerable for a few reasons. I don't know which tutorials you are talking about since you didn't reference any in the question. I cannot speak on behalf of those who wrote the unreferenced tutorials. They could just be wrong, or maybe the architecture of what you are trying to accomplish differs in some small way. I also don't know what framework or optional middleware you are using in your project.

Now, with all of that said there are a few things I can share that may help you out.

Most tutorials you find out there are probably not going to open a connection and read from the stream indefinitely. When the process ends, the http response is closed with .end() or something similar. Since an HTTP response is a write stream, it follows the same rules as any other stream. You can find a lot of good info about streams here:

https://github.com/substack/stream-handbook

Something important to understand is that a stream can have a buffer and most http frameworks enable compression which causes buffers to be used. The code sample in the next link is a good example of what a framework would do for you behind the scenes (a minimal implementation of course)

https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/#what-we-ve-got-so-far

Since you want the output to continue being updated, you either have to wait until the output buffer size is reached or you have to call .flush().

If you ARE using express, check out this next Stack Overflow post related to compression middleware. I believe you'll have to have it disabled for your /redis route.

Node Express Content-Length

I hope that helped a little. Like I said, its kind of hard to answer this question. ;)