当批处理大小很高时，JDBC应用程序会挂起批量插入

Question

I am trying to do bulk insert from linux server(say server X) using simple JDBC application.

When I try the application with batch size of 5, it is working fine.

But when I try it with batch size of 500, it is hanging. I am not getting any error or exception. When I query the database, I could see the 500 rows are inserted but the java application just hangs. The java application is not exiting.

The application just hangs at the following method call

try{
// register driver

// establish connection

// create statement
// add insert commands from inputFile to the batch
statemnt.executeBatch();  //---------------application hangs here
}
//catch block followed by finally block to close resources

I tried the same application in different linux server, I was able to successfully execute the application for batch size of 500, the application ran fine and exited normally.

The application is hanging only in server X and only when the batch size is high.

Then I checked the strace of the hung process in server X. I found that it is looping in the following system calls

$ strace -f -p [hung-java-process-id]

[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 882169546}) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 882258933}) = 0
[pid 104273] clock_gettime(CLOCK_REALTIME, {1417787058, 859159155}) = 0
[pid 104273] futex(0x7ffe78013754, FUTEX_WAIT_PRIVATE, 1, {0, 49972845}) = -1 ETIMEDOUT (Connection timed out)
[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 932601035}) = 0
[pid 104273] clock_gettime(CLOCK_MONOTONIC, {9635863, 932677888}) = 0
[pid 104273] clock_gettime(CLOCK_REALTIME, {1417787058, 909571241}) = 0
[pid 104273] futex(0x7ffe78013754, FUTEX_WAIT_PRIVATE, 1, {0, 49977759}) = -1 ETIMEDOUT (Connection timed out)
[pid 104273] futex(0x7ffe78013728, FUTEX_WAKE_PRIVATE, 1) = 0

I checked the list of open files opened by the hung java process and got the following output, which showed the db connection is still open.

$ lsof -p [hung-java-process-id] | grep TCP

COMMAND   PID       USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
java    104273 userid   14u  IPv6 1674188639      0t0  TCP serverX:55744->dbServer:blackjack (ESTABLISHED)

I also tried by setting connection AutoCommit property to false and then called commit after the executeBatch method. But still I am facing the same issue.

I also tried by splitting the batch with smaller batch size. The application is working fine upto batch size of 29 but when I increase the batch size to 30, it simply hangs without any error or exception.

I could not find the cause of the issue. If strace is showing the cause of the issue, then please let me know, what exactly does it mean and how to resolve it? Any help would be much useful. Thanks in advance.

Please note I am using Java 7 .

I suppose the issue could be caused by some configuration in the linux machine(server X) as the same application is working fine in other machines for large batch size. Any insights related to the machine configuration would be very useful for me.

Answer 1

which database is it? I saw some mentions of problems with postgresql drivers, I have no idea whether similar issues happen on other providers.

See in GitHub and PostGreSql