许多TIME_WAIT无法建立ESTABLISHED TCP连接

Question

I've got recently update system, and can't refresh context. It happens about 5 minutes after starting up Application Server, a tail of errors has

  "Invocation of init method failed; nested exception is org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]"

Then debugging Application Server or in other words a client of tcp-model.

netstat -a |grep 10.0.17.1|grep ESTABLISH 

shows port every time when stop debugging (about 3 items, must be more than 10), this ports keep established only if debug process stops at break point of a callback. In other hand there is dozens TIME_WAIT connections during refreshing.I've found a recommendation in http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html where tcp-client should reuse connections and make them not rapidly. The last one can be achieved by other database client. When open a new session, a tcp-connection is successfully ESTABLISHED and is kept as long as desire. For a short long life session tcp dump is

16:43:31.979499 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [S], seq 1105714340, win 29200, options [mss 1460,sackOK,TS val 4018590 ecr 0,nop,wscale 7], length 0
16:43:32.086237 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [S.], seq 1725221078, ack 1105714341, win 8192, options [mss 1410,nop,wscale 8,sackOK,TS val 1475034648 ecr 4018590], length 0
16:43:32.086281 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], ack 1, win 229, options [nop,nop,TS val 4018697 ecr 1475034648], length 0
16:43:32.086954 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 1:228, ack 1, win 229, options [nop,nop,TS val 4018698 ecr 1475034648], length 227
16:43:32.166135 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 1:9, ack 228, win 1026, options [nop,nop,TS val 1475034758 ecr 4018698], length 8
16:43:32.166192 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], ack 9, win 229, options [nop,nop,TS val 4018777 ecr 1475034758], length 0
16:43:32.166733 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 228:455, ack 9, win 229, options [nop,nop,TS val 4018777 ecr 1475034758], length 227
16:43:32.237317 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 9:41, ack 455, win 1025, options [nop,nop,TS val 1475034836 ecr 4018777], length 32
16:43:32.237604 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 455:614, ack 41, win 229, options [nop,nop,TS val 4018848 ecr 1475034836], length 159
16:43:32.311783 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 41:168, ack 614, win 1025, options [nop,nop,TS val 1475034909 ecr 4018848], length 127
16:43:32.312084 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 614:647, ack 168, win 229, options [nop,nop,TS val 4018923 ecr 1475034909], length 33
16:43:32.383071 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 168:407, ack 647, win 1025, options [nop,nop,TS val 1475034982 ecr 4018923], length 239
16:43:32.383744 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], seq 647:2045, ack 407, win 237, options [nop,nop,TS val 4018995 ecr 1475034982], length 1398
16:43:32.386640 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 2045:3232, ack 407, win 237, options [nop,nop,TS val 4018997 ecr 1475034982], length 1187
16:43:32.456443 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [.], ack 3232, win 1026, options [nop,nop,TS val 1475035056 ecr 4018995], length 0
16:43:32.457266 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [.], seq 407:1805, ack 3232, win 1026, options [nop,nop,TS val 1475035056 ecr 4018995], length 1398
16:43:32.458307 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 1805:2756, ack 3232, win 1026, options [nop,nop,TS val 1475035056 ecr 4018995], length 951
16:43:32.458402 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], ack 2756, win 281, options [nop,nop,TS val 4019069 ecr 1475035056], length 0
16:43:32.459388 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 3232:3399, ack 2756, win 281, options [nop,nop,TS val 4019070 ecr 1475035056], length 167
16:43:32.531250 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 2756:3022, ack 3399, win 1026, options [nop,nop,TS val 1475035130 ecr 4019070], length 266
16:43:32.532638 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 3399:4371, ack 3022, win 303, options [nop,nop,TS val 4019143 ecr 1475035130], length 972
16:43:32.616867 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 3022:3033, ack 4371, win 1022, options [nop,nop,TS val 1475035216 ecr 4019143], length 11
16:43:32.616899 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 3033:3044, ack 4371, win 1022, options [nop,nop,TS val 1475035216 ecr 4019143], length 11
16:43:32.616962 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], ack 3044, win 303, options [nop,nop,TS val 4019228 ecr 1475035216], length 0
16:43:32.617114 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 4371:4382, ack 3044, win 303, options [nop,nop,TS val 4019228 ecr 1475035216], length 11
16:43:32.688983 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [P.], seq 3044:3175, ack 4382, win 1022, options [nop,nop,TS val 1475035286 ecr 4019228], length 131
16:43:32.689413 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [P.], seq 4382:4392, ack 3175, win 325, options [nop,nop,TS val 4019300 ecr 1475035286], length 10
16:43:32.689465 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [F.], seq 4392, ack 3175, win 325, options [nop,nop,TS val 4019300 ecr 1475035286], length 0
16:43:32.758818 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [.], ack 4393, win 1022, options [nop,nop,TS val 1475035358 ecr 4019300], length 0
16:43:32.758874 IP 10.0.17.1.1521 > 192.168.88.246.39106: Flags [F.], seq 3175, ack 4393, win 1022, options [nop,nop,TS val 1475035358 ecr 4019300], length 0
16:43:32.758908 IP 192.168.88.246.39106 > 10.0.17.1.1521: Flags [.], ack 3176, win 325, options [nop,nop,TS val 4019370 ecr 1475035358], length 0

The main thing that the session from Spring keeps alive less than 1 second. There is no problem with opening ports and no problem with manually created database connection from other client as mentioned above.

Very, very need to discuss the issue!!!

Answer 1

Solve problem by merging backup /etc directory. The issue was in Linux. By comparing with nonworking case found that /proc/sys/net consists netfilter parameters. I'm not an expert in netfiltering so just try to avoid it. Before merging tried to remove some nf_* modules from kernel without success, no way to remove nf_conntrack module even with force key or blacklisting. Fortunately no need to rebuild kernel just merging /etc. The only thing that is different after merging is files mode, which kdiff3 doesn't show. Keeping entire one million possible solutions!