Jersey - Massive Latency Calling Service Method From Resource

Question

I have a Jersey REST service with a resource class that calls methods on a service class. During testing we've noticed a latency between the time the resource method's "Entering" log statement and the service's. This latency can be as much as 5 minutes although normally it's in the 2 minute range. Once in awhile, the latency is minimal (milliseconds).

Here's what our classes look like:

Resource

@Stateless
@Path("/provision")
public class ProvisionResource
{
    private final Logger logger = LoggerFactory.getLogger(ProvisionResource.class);

    @EJB
    private ProvisionService provisionService;

    @GET
    @Produces(MediaType.APPLICATION_XML)
    @Path("/subscriber")
    public SubscriberAccount querySubscriberAccount(
            @QueryParam("accountNum") String accountNum)
    {
        logger.debug("Entering querySubscriberAccount()");

        final SubscriberAccount account;

        try
        {
            account = provisionService.querySubscriber(accountNum);    
        }
        catch (IllegalArgumentException ex)
        {
            logger.error("Illegal argument while executing query for subscriber account",
                    ex);

            throw new WebApplicationException(Response.Status.BAD_REQUEST);
        }
        catch (Exception ex)
        {
            logger.error("Unexpected exception while executing query for subscriber account",
                    ex);

            throw new WebApplicationException(Response.Status.INTERNAL_SERVER_ERROR);
        }

        logger.debug("Exiting querySubscriberAccount()");

        return account;
    }    
}

Service:

@Singleton
public class ProvisionService
{
    private final Logger logger = LoggerFactory.getLogger(ProvisionService.class);

    public SubscriberAccount querySubscriber(final String accountNum) throws IllegalArgumentException, Exception
    {
        logger.debug("Entering querySubscriber()");

        if (null == accountNum)
        {
            throw new IllegalArgumentException("The argument {accountNum} must not be NULL");
        }

        SubscriberAccount subscriberAccount = null;

        try
        {
            // do stuff to get subscriber account
        }
        catch (Exception ex)
        {
            throw new Exception("Caught exception querying {accountNum}=["
                    + accountNum + "]", ex);
        }
        finally
        {
            logger.debug("Exiting querySubscriber()");
        }

        return subscriberAccount;
    }

Here's some samples from our logs showing the timestamps of when we enter the methods.

2012 Feb 07 15:31:06,303 MST [http-thread-pool-80(1)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount() 
2012 Feb 07 15:31:06,304 MST [http-thread-pool-80(1)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:35:06,359 MST [http-thread-pool-80(1)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:40:33,395 MST [http-thread-pool-80(1)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:34:06,345 MST [http-thread-pool-80(2)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:37:24,372 MST [http-thread-pool-80(2)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:33:06,332 MST [http-thread-pool-80(4)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:34:15,349 MST [http-thread-pool-80(4)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:37:24,371 MST [http-thread-pool-80(4)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:40:36,004 MST [http-thread-pool-80(4)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:32:06,317 MST [http-thread-pool-80(5)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:34:15,325 MST [http-thread-pool-80(5)] DEBUG my.package.ProvisionService - Entering querySubscriber()

2012 Feb 07 15:36:06,373 MST [http-thread-pool-80(5)] DEBUG my.package.ProvisionResource - Entering querySubscriberAccount()
2012 Feb 07 15:40:34,956 MST [http-thread-pool-80(5)] DEBUG my.package.ProvisionService - Entering querySubscriber()

As you can see, the first one called the querySubscriber method in the service almost immediately after entering the resource's querySubscriberAccount. However, subsequent calls to the webservice take ~1 to 5 minutes. There's really nothing happening in the resource that would hold up processing/calling the service.

The webservice is deployed on a Linux server in Glassfish 3.1.1.

Has anyone seen anything like this before?? Any suggestions on what's going on?

EDIT

Just a little more information...

The domain into which the web service war is deployed has 4 applications deployed in it:

• the Jersey webservice war which is having issues
• an ear which uses a client to the Jersey webservice
• a servlet war used to test connections etc. used by the ear (including the Jersey webservice)
• another servlet war which does not use the webservice

When we disabled the ear and "other" war file (only the Jersey war and the test servlet were enabled), the latency issue goes away. We re-enabled the war and ear and things still continued to respond in a timely manner. When we redeployed Jersey webservice war (made some logging changes), the latency problem immediately came back.

Answer 1

Thread dump can be used to find out what code (including stack traces) is running and the current moment inside Java process. jps tool will help in get getting PID of the required JVM instance.