Weblogic类加载器中的线程被卡住

Question

We have a situation here where more than 100 threads are stuck in a weblogic (10.3.6) managed server.

I am analysing a threaddump and find 150 threads blocked waiting on a lock in the classloader, like : "jmsContainer-14" prio=10 tid=0x00007f11485d7000 nid=0x14c8 waiting for monitor entry [0x00007f1023590000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48) - waiting to lock <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader) at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:207)

(10 occurences)

or

"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f10d002d800 nid=0x5ea5 waiting for monitor entry [0x00007f1136eeb000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48) - waiting to lock <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader) at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115) at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)

(137) occurences

No all stuck thread are blocked for the same reason, but we have 150 threads waiting to lock the same lock <0x0000000782aeb288>, wich I find strange because it is in the classLoader.

I don't think it is the root cause of the problem we are experiencing right now, but if someone could explain this behavious it could be of great help.

Thanks

EDIT :

Here is the blocking thread :

    ExecuteThread '136' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f11484af800 nid=0x7640 runnable [0x00007f102c11b000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:632)
    at java.util.zip.ZipCoder.getBytes(ZipCoder.java:86)
    at java.util.zip.ZipFile.getEntry(ZipFile.java:306)
    - locked <0x0000000782b2ddf8> (a java.util.jar.JarFile)
    at java.util.jar.JarFile.getEntry(JarFile.java:226)
    at weblogic.utils.classloaders.ZipClassFinder.getSource(ZipClassFinder.java:39)
    at weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:50)
    at weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:31)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.application.utils.CompositeWebAppFinder.getClassSource(CompositeWebAppFinder.java:88)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
    at weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:25)
    at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:291)
    - eliminated <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
    at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
    at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
    at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:49)
    - locked <0x0000000782aeb288> (a weblogic.utils.classloaders.ChangeAwareClassLoader)
    at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115)
    at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)

sorry, I should have posted it earlier.

Nearly all the threads are bloked in, or blocked by one of two lines of codes containing this piece of code:

DatatypeFactory.newInstance().newXMLGregorianCalendar(dateDebCre)

Apparently Weblogic is trying very hard to load the same jar again and again. Which I do not understand, because it is supposed to be stored in a cache in memory, right ?

First, we will remove this lines of code as they are completely useless (a shame it is). But again, I think understanding the problem would be better.

Thank you

Answer 1

First you need to find out who is holding lock <0x0000000782aeb288>. If possible attach complete thread dump so that I can have a look.

Answer 2

From the thread dump it also looks like source code is trying to read data from zip file. If the size of the files are huge then it can lead to potential issue If you are running on linux then you can check in /var/proc directory if the process is doing a lot of IO