群集生产服务器中的性能问题

Question

Our production users complain of performance issue at least two to three times every month. We have IBM WAS 8 servers in production. The application uses two SOAP based services say H and T. H is deployed on INTERNET clustered servers (X, Y). T is deployed on INTRANET servers (U, V). Client directly connects to H. H connects to T on INTRANET . Both the SOAP based services H,T connects to a database. Also, there is a service for Authenticating users. We are not seeing any errors in the logs of server U and V. But logs of H on server X, Y gives following error. Different error at different times:

1. java.net.SocketTimeoutException: Socket operation timed out before it could be completed
2. java.io.IOException: Connection close: Read failed.  Possible end of stream encountered.  
java.lang.OutOfMemoryError: GC overhead limit exceeded
3. Exception - User fault processing is not supported. The @WebFault faultbean is missing for java.rmi.RemoteException
4. Authentication failed

We are thinking of increasing the heap size. But, before doing that what performance parameters we should collect from server to narrow root cause of issue

Answer 1

As a first step, you should always monitor the underlying system (hardware server, VM, container) key performance resources - CPU utilization, free memory, network usage, etc. If your box is running out of CPU cycles or free RAM, app server performance will suffer.

As the next layer, there are various performance metrics provided by Java and by WAS which can help diagnose an issue like yours. A useful guide to WAS performance investigation is the WebSphere Application Server Performance Cookbook https://publib.boulder.ibm.com/httpserv/cookbook/
In your case probably this section is most applicable: https://publib.boulder.ibm.com/httpserv/cookbook/Recipes-WAS_Traditional_Recipes-General_WAS_Traditional_Performance_Problem.html

One of the errors in your list is an OOM thrown due to "GC overhead limit exceeded". This means that the server JVM ran critically low on free space in the java heap, so that it was spending almost all its time running Java garbage collection trying to free space to do real work. This type of problem can cause other problems you listed, such as timeouts and communication failures.

To diagnose an excessive GC issue, you need verbose GC logging - enabling verbose GC is step #2 in the second link above, also explained at http://www-01.ibm.com/support/docview.wss?uid=swg21114927 Verbose GC logging is very low overhead and has very high diagnostic value, so it should be enabled at all times, including in production environments.

The most critical information from the GC log is how much free tenure heap is available after each global GC. This should be at least 30% of the total tenure heap size, or the JVM will have to do too much GC work to clear space for the 'real work' you want your server to perform. The "GC overhead limit exceeded" error typically arises in configs when there is less than 10% free tenure space on a busy server.

If a server is consistently running at less than 30% free tenure space after global GC, you need to either increase the heap size or shift some workload off the server.