Performance issue in production

Question

Hi I am working on a web based java tool. We have two instances of our tool deployed in 2 separate servers – one in production (used concurrently by many customers) and one in dev environment rarely used by me. Both the instances, dev and production, are deployed in exactly equivalent servers with same hardware specifications, have the exact code, heap size size, application server, version etc.

But I am observing much slowness in the production instance, compared to dev instance even at odd hours in the night when customer usage is zero (We have tools to monitor customer usage in production). This is surprising because all factors are identical. I am assuming that since the production instance is highly used, the heap there is almost full and fragmented while the heap in dev is relatively free and unfragmented. This is why the dev instance is much faster, even though everything is identical. Is this a reasonable assumption since a fragmented almost full heap will spend more time in object creation and thus degrade performance?

Answer 1

Most GC perform copies and compaction which avoid any fragmentation. Only Concurrent Mark Sweep can get some fragmentation, however this shouldn't be a problem in most cases.

Rather than guessing what your problem could be I suggest you measure what the system is doing. Ideally you want to profile your production system. eg during the night or under low load. eg VisualVM (free) or YourKit (better, not free,) If this is not an option. you can monitor its memory usage and how much time it spends GCing. eg with jstat

However, if you haven't profiled your application, I suggest you do so because there are likely to be some obvious and simple things you can do to improve performance, even if you run it on development with a reasonably realistic work load.

Are you sure you servers is not being scanned by web scanners? (They tend not to sleep and can be active any time of the day/night)

Answer 2

Why was it assumed that the heap is fragmented? You can use VisualGc, a plugin for visual VM to analyze GC stats in real time. You can take one of the production servers out of traffic and profile it too.

http://java.sun.com/performance/jvmstat/visualgc.html

There are many factors to consider in your analysis. Latency, Network / Disk IO, database records and their size + indexes, etc to name a few.