Please find below the application/environment details where the problem is seen.
I see lot of threads in the application are getting into a BLOCKED state sometimes for few mins. On thread dump analysis, it was found that java.net.InetAddress.getLocalHost call is taking too much time. Lot of threads are getting stuck here. The host name is fetched for every logger printed in the application.
The issue is intermittent. But when it occurs, the application/tomcat will go into a paused state which leads to the accumulation of lot of threads. After some time(few seconds), all the blocked threads are unblocked simultaneously. Because of the request concurrency, the application will run out of DB connections which it maintains in the pool leading to issues/slowness/service availability. As a fix, I have made sure to access the host name only once into a static variable and use the same throughout the logging process. I wanted to know the detailed root cause of this issue.
Sample below from the thread dump:
"https-jsse-nio-8443-exec-13" #95 daemon prio=5 os_prio=0 tid=0x00007fccadbba800 nid=0xaf5 waiting for monitor entry 0x00007fcb912d1000
java.lang.Thread.State: BLOCKED (on object monitor)
at java.net.InetAddress.getLocalHost(InetAddress.java:1486)
- waiting to lock <0x00000005e71878a0> (a java.lang.Object)
In JDK 8, InetAddress.getLocalHost()
works as follows :
Steps 2-4 are performed under the global cacheLock
. If something goes wrong during this process, all threads calling InetAddress.getLocalHost()
will block at this lock - exactly what you observe.
Usually local host name resolution does not end up in a network call, as long as the host address is hard-coded in /etc/hosts
. But in your case it seems like the real network requests are involved (whenever TTL expires). And when the first DNS request times out (UDP is not a reliable protocol after all), a delay happens.
The solution is to configure /etc/hosts
to contain the name and the address of the local host, eg
192.168.1.23 myhost.mydomain
where myhost.mydomain
is the same string as returned by hostname
command.
Finally, if the host name is not expected to change while the application is running, caching it once and forever on the application level looks like a good fix.
To fix the issue, I am loading the hostname only once and caching it during the application start up. I have rolled out this fix to production and we are not seeing the thread blocking issues anymore.
Maybe server is going to look using ipv6 and if is not in use you can configure JVM to use only IPV4, to do so add this to the options -Djava.net.preferIPv4Stack=true or if only need ipv6 -Djava.net.preferIPv6Stack=true. This will force JVM to use the right protocol.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.