How to monitor memory usage and build scalable java web server?

Question

We are writing a web application with Java, which is served as chat room server powered by WebSocket. We design the server structure properly so that we can run multiple identical server instance to support massive connection and communication. The problem is that we can't scale the servers cluster dynamically. Ideally, any slave node should monitor its memory usage and notify the dedicated master node when it can't accept more connection. Now the web application simply crashes when too much connections come in rapidly. Are there any solution (or idea) to improve the situation?

Answer 1

In java its fairly easy to check up some information about the current cluster.

MemoryUsage heapMemoryUsage = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();
heapMemoryUsage.getUsed();

If you want to check the JVM Runtime total memory for that cluster, here`s an example on how to do that.

public class TestMemory {

    public static void main(String [] args) {

        int mb = 1024*1024;

        //Getting the runtime reference from system
        Runtime runtime = Runtime.getRuntime();

        System.out.println("##### Heap utilization statistics [MB] #####");

        //Print used memory
        System.out.println("Used Memory:" 
            + (runtime.totalMemory() - runtime.freeMemory()) / mb);

        //Print free memory
        System.out.println("Free Memory:" 
            + runtime.freeMemory() / mb);

        //Print total available memory
        System.out.println("Total Memory:" + runtime.totalMemory() / mb);

        //Print Maximum available memory
        System.out.println("Max Memory:" + runtime.maxMemory() / mb);
    }
}

So, now any child nodes know whey they are on theyr limit. If you have a open WebSocket comunication, you should have a ping on all child nodes and check theyr status, if they are heavily loaded you could instantiate a new node dynamically, for example, using Docker to make things easyer. (just deploy a new Docker container on a cloud based server, connect to the master node (you could use Node.js to connect via WebSocket to the master node eventually, so you could even connect before your JVM is actually on.) just telling "hey, new node here, gimme some stuff to do". Internally i would not know how far redirecting traffic from one node to other would be, but i hope this idea. If you want a more accurate connection (fewer crashes) dont only look at memory usage, but network usage and if any TCP packages are lost or websocket gets suddenly closed, the server should tell the master node right away to create a new instance of it.

Answer 2

Try utilizing a tool like JProfiler or IntelliJ for noticing how your application behaves in overloaded situations, also it can help you in noticing if you ave some threads which aren't garbage collected and so much more. A tool like the ones mentioned above can check the load on your webserver or jvm, however your application is deployed to identify if there's any need to expand hardware or just an application correction would suffice.

A tool to overload the application would be Jmeter that'd help you do the stress testing for long periods of times.