weblogic无法处理任何请求

Question

Recently our weblogic hangs frequently after it was started for some hours,when it hangs,it can't process any request。 We saw the thread dump,most of the threads were doing this:

 "[ACTIVE] ExecuteThread: '195' for queue: 'weblogic.kernel.Default (self-tuning)'" - Thread t@5932
   java.lang.Thread.State: RUNNABLE
        at java.util.Hashtable.get(Hashtable.java:335)
        at java.util.Properties.getProperty(Properties.java:774)
        at java.lang.System.getProperty(System.java:630)
        at oracle.net.nl.NVPair.<init>(Unknown Source)
        at oracle.net.nl.NVFactory._readNVPair(Unknown Source)
        at oracle.net.nl.NVFactory._readNVList(Unknown Source)
        at oracle.net.nl.NVFactory._readNVList(Unknown Source)
        at oracle.net.nl.NVFactory._readNVList(Unknown Source)
        at oracle.net.nl.NVFactory._readNVPair(Unknown Source)
        at oracle.net.nl.NVFactory.createNVPair(Unknown Source)
        at oracle.net.nt.TcpNTAdapter.<init>(Unknown Source)
        at oracle.net.nt.ConnOption.getNT(Unknown Source)
        at oracle.net.nt.ConnOption.connect(Unknown Source)
        at oracle.net.nt.ConnStrategy.execute(Unknown Source)
        at oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
        at oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
        at oracle.net.ns.NSProtocol.connect(Unknown Source)
        at oracle.jdbc.ttc7.TTC7Protocol.connect(TTC7Protocol.java:1774)
        at oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:215)
        at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:362)
        at oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:536)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:328)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:300)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:204)
        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1057)
        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:982)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:371)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:294)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:284)
        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:466)
        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:357)
        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:83)
        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:96)
        at weblogic.jdbc.pool.Driver.connect(Driver.java:150)
        at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:647)
        at weblogic.jdbc.jts.Driver.connect(Driver.java:137)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
        at com.xx.core.database.Database.connect(Database.java:187)
        at com.xx.core.database.Database.connect(Database.java:148)
        at com.xx.core.database.Database.newInstance(Database.java:816)
        at com.xx.DatabaseProvider.getDatabase(DatabaseProvider.java:10)
        at com.xx.Tools.getDatabase(Tools.java:91)
        at com.xx.PopInfo.doGet(PopInfo.java:179)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:225)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:127)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
        at com.anychart.servlet.ResourceFilter.doFilter(ResourceFilter.java:53)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3212)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:1983)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1890)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1344)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

Through this stacktrace,it seems all these threads was waiting for getting database connection , but when we saw the oracle,we saw a lot of inactive jdbc connection。
Why so many threads are waiting for connection but at the same time database has so many inactie jdbc connection, how can we solve this problem, please help us ,thanks.

Answer 1

From your stack trace it looks they arestuck due to deadlock while accessing system property. Can you please change the jdbc driver and try. I'm not sure from which jar file the class oracle.net.nl.NVPair is used. When I checked source code of oracle.net.nl.NVPair from couple of jars the only reference to system property I see is private String _linesep = System.getProperty("line.separator"); . This is matching with your stack trace at oracle.net.nl.NVPair.(Unknown Source)