Java threads and HTTP requests deadlock

Question

I encounter a problem with java threads and I am not sure if its related to my approach or if thread pooling is going to resolve what I am trying to achieve.

        for (int i = 0; i < 100; i++) {
            verifier[i]=new Thread();
            verifier[i].start();
        }

I initialize 100 threads and start them. In the threads the code that gets executed is just

        con = (HttpURLConnection) website.openConnection(url);
        // gets only the header
        con.setRequestMethod("HEAD");
        con.setConnectTimeout(2000); // set timeout to 2 seconds

These threads repeat the process above over a long list of url/data.

The first 50 threads execute almost instantly then they just stop for 60 seconds or so and then there is another spike of execution that 20 of them or so finish at the same time and so on. The same deadlock occurs even if there is 4 of them.

My first guess was a deadlock. I am not sure how to resolve the issue and maintain a constant execution pace, without deadlocks and stops.

I am looking for an explanation of why this occurs and how it can be resolved.

By DeadLock I reefer to the Java Virtual Machine and how it handles thread. Not deadlock caused by my threads.

SCREENSHOT OF THREAD EXECUTION:

It looks like the threads are dying for no reason and I don't know why?!

Answer 1

It could be that the operating system configurable limit of tcp/ip connections gets hit, which causes the JVM to block waiting for a new TCP/IP connection to be created, which will only happen if a connection already used get's closed.

This could help to find what is going on:

• profile the run with visualvm which comes with the JVM itself (run it on the command line with jvisualvm). There should be indication of how many threads are created and why are they blocked, deadlocks, etc.

• Wait for it to block and take thread dumps of the JVM process to check for deadlocks in the thread stack traces using jstack or visualvm, search for the deadlock keyword.

• Check with netstat -nao the state of your TCP connections, to see if the operating system limit is getting hit, if there are many connections in CLOSE_WAIT at the times the blocking occurs

• Are you behind a corporate proxy/firewall, you could be hitting some other sort of security limit that prevents you from opening more TCP connections, not necessarily the limit of the operating system

If none of this helps you can always edit the question with further findinds, but based on the description of the code other limits are getting hit that on a first look don't seem related to JVM thread deadlocks, hope this helps.