GuavaCache expireAfterWrite（）导致过多的阻塞线程

Question

We have a use case where our data is becoming stale after some time. Hence we chose expireAfterWrite cache eviction strategy. However this is causing too many blocked threads in threadpool. Any idea why this might be happening and if there is any way to avoid this.

sample thread pool log:

Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
Deadlock Detection:

No deadlocks found.

Thread 7546: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
 - java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
 - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)

Locked ownable synchronizers:
    - None

Thread 1268: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - com.amazon.coral.google.common.util.concurrent.AbstractFuture$Sync.get() @bci=2 (Compiled frame)
 - com.amazon.coral.google.common.util.concurrent.AbstractFuture.get() @bci=4 (Compiled frame)
 - com.amazon.coral.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(java.util.concurrent.Future) @bci=3 (Compiled frame)
 - com.amazon.coral.google.common.cache.LocalCache$LoadingValueReference.waitForValue() @bci=4 (Interpreted frame)
 - com.amazon.coral.google.common.cache.LocalCache$Segment.waitForLoadingValue(com.amazon.coral.google.common.cache.LocalCache$ReferenceEntry, java.lang.Object, com.amazon.coral.google.common.cache.LocalCache$ValueReference) @bci=44 (Interpreted frame)
 - com.amazon.coral.google.common.cache.LocalCache$Segment.get(java.lang.Object, int, com.amazon.coral.google.common.cache.CacheLoader) @bci=122 (Interpreted frame)
 - com.amazon.coral.google.common.cache.LocalCache.get(java.lang.Object, com.amazon.coral.google.common.cache.CacheLoader) @bci=17 (Compiled frame)
 - com.amazon.coral.google.common.cache.LocalCache.getOrLoad(java.lang.Object) @bci=6 (Compiled frame)
 - com.amazon.coral.google.common.cache.LocalCache$LocalLoadingCache.get(java.lang.Object) @bci=5 (Compiled frame)
 - com.amazon.csapp.services.SmartVideoCacheService.getVideos(com.amazon.csapp.utils.MetricsContext, com.amazon.csapp.utils.AmazonRequestInfo) @bci=33, line=214 (Compiled frame)
 - com.amazon.csapp.rest.resource.v2.VideoResource.getVideos(java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.String) @bci=67, line=56 (Interpreted frame)
 - sun.reflect.GeneratedMethodAccessor90.invoke(java.lang.Object, java.lang.Object[]) @bci=88 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
 - org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=3, line=81 (Interpreted frame)
 - org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(java.lang.Object, java.lang.Object[]) @bci=39, line=128 (Interpreted frame)
 - org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(java.lang.Object, javax.ws.rs.core.Request) @bci=6, line=195 (Interpreted frame)
 - org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(java.lang.Object, org.glassfish.jersey.server.ContainerRequest) @bci=3, line=94 (Compiled frame)
 - org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(org.glassfish.jersey.server.ContainerRequest, java.lang.Object) @bci=13, line=353 (Compiled frame)
 - org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(org.glassfish.jersey.server.ContainerRequest) @bci=103, line=343 (Interpreted frame)
 - org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(java.lang.Object) @bci=5, line=102 (Interpreted frame)
 - org.glassfish.jersey.server.ServerRuntime$1.run() @bci=198, line=237 (Compiled frame)
 - org.glassfish.jersey.internal.Errors$1.call() @bci=4, line=271 (Compiled frame)
 - org.glassfish.jersey.internal.Errors$1.call() @bci=1, line=267 (Compiled frame)
 - org.glassfish.jersey.internal.Errors.process(java.util.concurrent.Callable, boolean) @bci=36, line=315 (Compiled frame)
 - org.glassfish.jersey.internal.Errors.process(org.glassfish.jersey.internal.util.Producer, boolean) @bci=2, line=297 (Compiled frame)
 - org.glassfish.jersey.internal.Errors.process(java.lang.Runnable) @bci=9, line=267 (Compiled frame)
 - org.glassfish.jersey.process.internal.RequestScope.runInScope(java.lang.Runnable) @bci=25, line=322 (Interpreted frame)
 - org.glassfish.jersey.server.ServerRuntime.process(org.glassfish.jersey.server.ContainerRequest) @bci=22, line=211 (Interpreted frame)
 - org.glassfish.jersey.server.ApplicationHandler.handle(org.glassfish.jersey.server.ContainerRequest) @bci=5, line=979 (Compiled frame)
 - org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=115, line=344 (Compiled frame)
 - org.glassfish.jersey.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=372 (Compiled frame)
 - org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=321, line=335 (Compiled frame)
 - com.amazon.csapp.servlet.JerseyServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=15, line=26 (Compiled frame)
 - org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=39, line=218 (Compiled frame)
 - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=376, line=290 (Compiled frame)
 - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)
 - org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=804, line=233 (Compiled frame)
 - org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=365, line=191 (Compiled frame)
 - org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=327, line=470 (Compiled frame)
 - org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=64, line=127 (Compiled frame)
 - org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=6, line=103 (Compiled frame)
 - com.amazon.tomcat.valves.QueryLogValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=47, line=78 (Compiled frame)
 - org.apache.catalina.valves.AccessLogValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=24, line=617 (Compiled frame)
 - org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=42, line=109 (Compiled frame)
 - org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response) @bci=158, line=293 (Compiled frame)
 - org.apache.coyote.http11.Http11Processor.process(java.net.Socket) @bci=545, line=859 (Compiled frame)
 - org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket) @bci=73, line=610 (Compiled frame)
 - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=503 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)

Answer 1

You should probably add a dedicated thread to refresh the cache's values more frequently. As Ben Manes suggests configuring a refreshAfterWrite that's lower than your expiration threshold would also help, but you'll want to override reload() to do the work asynchronously.

It's also possible there are hotspots in your code causing numerous separate paths to try to hit the same cache keys simultaneously. If you don't expect that behavior (eg you expected approximately one thread at a time to be accessing a key) you might want to investigate whether there's some missing locking semantics causing too many threads to be reading from the cache at the same time, or a different cause of hotspots such as unusual request patterns.