How Can I or Should I Fine Tune Node.js/Socket.io Chat App So It Doesn't Drag Down My Entire Website/Server?

Question

I built a chatroom w/ NodeJs using, socket.io, mysql2 & express packages. It's extremely dynamic including public chats, private messaging, and other unique features to make it lively, ie real time updates, notifications, etc.

Originally built and tested on a dummy domain and was working perfectly fine. And then I moved it over to my main website domain when finished. Now that it is, I'm experiencing horrible site load speeds. However, both domains (the test domain site and main live site were on the same server).

The server; 2x Intel® Xeon® Processor E5-2697 v4, 64GB RAM, 4x 480GB SSD, HW RAID storage, Network port 1 Gbps, 2 ipv4 addresses, bandwidth 50 TB, Centos 7, cpanel included and have. About 1000 users actively use the website/chat. The chatroom is negatively impacting the rest of the site and slowing it all down. Initial server response is roughly 14 seconds, but in worst case scenarios has been as long as 30+ seconds. Whereas on the test domain it was only literally 1 second. It goes without saying I should expect as good a load times as on the original test domain as a live one with real users, but this bad? I digress.

I ran speed tests. The results of those tests kept reporting back to me the initial load time was too long and literally said nothing in specific to point me to what the problem was so I assumed it was MYSQL, php, among other things. I optimized it all to the best of my ability. But after the site kept dragging, I still kept getting the same results. I finally removed things from the site one by one and when I removed the chat, the site loaded perfectly fine. It was odd, because none of the speed tests pointed to the chat or node in particular being the reasoning.

So then I optimized Node in the following ways. -I tried adding pooling to the sql connection, trying different numbers, from 5-500, -I tried adding a PoolManager to the database connection. -I removed the ability for all visitors to stay connected to our node server. They're socket connection would be terminated if they didn't mean certain cretia. -I added some parallelize to the code. To run some functions that called the database. -I made sure to add keys and avoided long sql queries, used LIMIT, and avoided *

Put it back on the site, still the same horrible load times.

After some research I am hopeful (but still only guessing) that the following may be cause for concern;

-The server is defaulting to polling. So maybe that might be causing an issue. Since I assume everytime it polls it has to connect and disconnect from the database server which will cause a pretty big delay since I read that a lot of issue are there. -I tried to fixing the polling but have not been able to get it to work. I tried forcing transport:

['websocket']

, with all different types of settings, CORS, secure, and etc.

I read into possibly using Redis for caching and I think it can be implemented in some areas for sure. But with how lively the chat is. Cache might be limited in how often it can be used.

If there are any problems with my own code I might guess it is here.

//server
const f = require('./functions.js');
const db = require('./db.js');
const cookie = require('cookie');
const http = require("http");
const express = require("express");
const { Server } = require("socket.io");


const port = 3000;
const appBaseUrl = "/node";
const socketBaseUrl = "/node/socket";

global.last_message_time = new Date('1995-12-17T03:24:00');


// Create the server and socket
const expressApp = express();
const nodeServer = http.createServer(expressApp);
const socketIo = new Server(nodeServer, {
    path: socketBaseUrl,
});


//client 
const chat = io.connect("domain.com", {path: "/node/socket"});

But at the end of the day I am still at a loss and so I look to you and throw myself at the mercy of StackOverflow. Opinions, feedback and ideas as to why the chat is not playing nicely with my website would be more appreciated than you know. Thank you in advance and I look forward to hearing from you.

------------------------------Update

@Gaëtan Boyals - the additional code you asked for concerning where messages are handled, below.

let chats = []
            await db.getMessages(data.id, user.user_id, user.isAdmin, 'user')
            .then(response=>{
                chats = response.reverse();
            }).catch((err)=>{
                console.log(err);
            })

            let chats_html = [];
            for(let chat of chats){
                chat.message_type = chat.type;
                chat.images       = [];
                chat.small        = [];
                if(chat.type=='photos'){
                    let encoded = JSON.parse(chat.message);
                    for(let small of encoded){
                        let j = f.createToken(small);
                        chat.small.push(j+ "?height=150");
                        chat.images.push(j)
;
                    }
                }
                //might cause issues later so see what's up here.
                
                chat.emojis       = await db.getEmojiReactions(1, chat.id);
                chat.avatar       = f.createToken(chat.avatar);
                chats_html.push(f.createMessage(chat, user.user_id, user.user_type)); 
            }
            let chat_info = await db.getPrivateRoom(data.id, user.user_id);
            socket.join(chat_info);
            user.current_room = data.id;
            user.current_type = 'user';
            user.room = chat_info;

            let send = {chats:chats_html};
            socketIo.to(socket.id).emit('get-chat', send);

            let best_badge = await db.bestBadge(data.id);
            let user_1     = await db.getUserFromId(data.id);
            user_1.badge   = best_badge ? best_badge.badge_id:false;
            user_1.avatar  = f.createToken(user_1.avatar);
            
            let group_user_html = [];
            group_user_html.push(f.rightSideUsers(user_1));
            group_user_html.push(f.rightSideUsers(user));
            socketIo.to(socket.id).emit('right-users', group_user_html);

if(user.current_room != 0){
            let isMuted = false;
            if(data.type=='group'){
                isMuted = await db.isMuted(user.user_id, data.id);
            }
            if(!isMuted){
                data.user_id = user.user_id;
                data.to      = data.id;
                if(data.message_type == 'photos'){
                    data.message = JSON.stringify(data.message);
                }
                
                if(user.user_id==0){
                    data.guest = user.username;
                }
                db.newMessage(data)
                .then((response)=>{
                    if(response){
                        
                        if(data.type=='group'){
                            db.setChatMeta(response, 'group', data.id,user.user_id);
                        }else{
                            db.setChatMeta(response, 'user', data.id,user.room);
                        }
                        

                        data.id      = response;
                        data.chat_id = response;
                        let html     = f.createMessage(data, user.user_id, user.user_type);
                         
                        let new_data = f.createRightData(data);
                        new_data.message = new_data.message.replace('<br />', '');
                        let uni = user.room;
                        let send = {html:html, id:uni,type:data.type}
                        if(data.message_type == 'video'){
                            setTimeout(()=>{
                                socketIo.in(user.room).emit("new-message",send);
                                socketIo.emit('refresh');

                            }, 5000)
                            
                        }else{
                            console.log(user.room);
                            socketIo.in(user.room).emit("new-message",send);
                            socketIo.emit('refresh');

                        }
                        
                    }
                }).catch((err)=>{
                    console.log(err)
                })
            }else{
                socketIo.to(socket.id).emit('muted')
            }
        }

GET and SET messages.

Answer 1

Your 12-core server is not very large at all, at least when combining multiple applications. Your mileage may vary and greatly in some cases if the database was using all 12-cores and the Node.js app is using 1 core but gets bumped a lot the Node.js app suffers.

Now I don't know how large your site is how many users hit the site, but 12-cores of power is a lot of power...

1 core of power my HTTP server can handle a million request an hour with "many timeouts" this though would be hitting a single core database on a different server. So if we do the math here, that's quite a lot of requests we got there and you can benchmark to see yourself.

---

With all that said, you shouldn't have a problem. More detail would be required and I can edit this.

Answer 2

As far as I can tell going through that I believe somehow your server is filling to create web socket connections and use polling as a fallback. Which basically means a huge overhead. Can you perhaps separate the chat functionality to a separate node application and possibly containerize it?. Make the initial site load and then load the chat feature until you figure out what exactly is the root cause for this.

Answer 3

The user @S1ckhead already partly said what I'm about to explain, but I'll try to go more into details.

As I said in the comments, keep in mind that it's what I think is happening, but then again, @S1ckhead seems to have roughly the same idea as me so you might want to read on, even if it won't provide a full-baked solution.

This is not a hardware-related problem

Of course, we can brag about our personal set-ups and since we're at it, if I can run a self-hosted, bare-metal K8s cluster with three 2006 Dell PCs handling a Redis instance, a MongoDB instance and about 30 micro-services, I think the OP can run a PHP site, a NodeJS service and a MySQL instance with the specs he gave in his question just fine .

He mentioned contacting his host about the issue and asked them if he should upscale, they said no and I think they are the most qualified people to say so.

Also, imagine running only an instance of anything (Php, MySQL, you name it) per server... We're not all Bill Gates, and there's no need for us to be.

Reverse-proxy? Routing?

I will assume you have basic or intermediate level when it comes to Sys Admin / Networking, since you know how to set-up domains and link it to your server.

A reverse proxy is basically a software that routes the client request to specific applications. For example, let's say you have 3 domains (domainA.com, domainB.com, domainC.com), all pointing to the same IP address (so basically, the same machine). How does the server know that domainA should hit the PHP site, and domainB should hit a NodeJS service? That's where the reverse proxy comes in handy. That is very roughly explained, but you can find more explanation here and if you are like me and prefer graphic representations, here is one:

Now I'm not going to lie, I am not familiar at all with cPanel, PHP and such, I usually go the no-admin-tools route and host NodeJS-based apps, but you can surely set-up a reverse proxy with your set-up.

What is the real problem

Again, mentioning @S1ckhead, Socket.IO is probably using long-polling as a default, but why?

Since in your code, you make socketIO listen on your main domain: const chat = io.connect("domain.com", {path: "/node/socket"}); , the WebSocket requests will probably go to your PHP app. But your PHP app is probably not configured to handle WebSockets, so it defaults to long-polling. You can read more here , but basically, stating this site:

The server holds the request open until new data is available. Once available, the server responds and sends the new information.

So since it holds the request until something new happens, it could explain why you experience this extremely laggy feeling.

What should you do?

Well, you could go two routes, the first one and the easiest would be to loan a new server, buy another domain or subdomain, link the two together, host your NodeJS app here and link your main site to this new domain/subdomain.

The other, greatly reducing costs, would be to setup a reverse proxy as previously mentioned. It could prove to be a difficult challenge (I did it with no mentor/previous experience myself, but I also had no work obligations/pressure to do it) however. It is up to you and your constraints/requirements to chose which one you want to follow.

Of course, other problems will arise when your pool of user grow larger, and the question of scaling will make sense, but that's not the concern right now.

A final note on Socket.IO

Keep in mind that Socket.IO is a wrapper around the Web Socket protocol , and that it will only work with its client/server counterpart (meaning you can't wire up your own WebSockets client implementation with a Socket.IO server).

While I dislike the fact that you can't separate Socket.IO's client from its server, I also like the level of abstraction it provides. Of course, you could go with your own implementation if you're up to the challenge, but it's in my opinion not worth it as it will not improve the performances (and can, in fact, worsen them if the implementation is poorly written) by that much .