ExecutorCompletionService not taking an item returned by Callable?

Question

I'm having strange behavior from the ExecutorCompletionService. The item gets added to the ExecutorCompletionService.submit() fine. Then it gets worked on and is returned by the Callable worker thread which was previously submitted. After that return the ExecutorCompletionService.take() never sees it so never sees the blocking to return anymore items? I'm really not sure what is going on. I have created print lines and can see it completing the Callable worker thread. As soon as that happens the ExecutorCompletionService.take should be ready to take but in some cases the thing locks up and sometimes its fine?

I have created a test case if you run it a few times you will see it will in some cases lock up and never take any of the finished threads

ThreadDeadlockDemo

import java.util.Observable;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Callable;
import java.util.concurrent.CompletionService;
 import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorCompletionService;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.LinkedBlockingQueue;

public class ThreadDeadlockDemo extends Observable implements Runnable  {

private CompletionService<String> pool;
private ExecutorService executor ;
private Thread responseWorkerThread;
private HttpSchedulerWorker schedulerWorker;
private boolean shouldRun = true;
private int numThreadsInPool;
private BlockingQueue<String> queue;
public ThreadDeadlockDemo(int numThreads)
{
    numThreadsInPool = numThreads;
    executor = Executors.newFixedThreadPool(numThreads);
    pool = new ExecutorCompletionService<String>(executor);
    schedulerWorker = new HttpSchedulerWorker();
    responseWorkerThread = new Thread(schedulerWorker);
    responseWorkerThread.start();
    queue = new LinkedBlockingQueue<String>();
    new Thread(this).start();
}

public ThreadDeadlockDemo()
{
    numThreadsInPool = 1;
    executor = Executors.newFixedThreadPool(1);
    pool = new ExecutorCompletionService<String>(executor);
    schedulerWorker = new HttpSchedulerWorker();
    responseWorkerThread = new Thread(schedulerWorker);
    responseWorkerThread.start();
    queue = new LinkedBlockingQueue<String>();
    new Thread(this).start();
}

public void setThreadCount(int numThreads)
{
    executor = Executors.newFixedThreadPool(numThreads);
    pool = new ExecutorCompletionService<String>(executor);
    numThreadsInPool = numThreads;
}

public void add(String info) {
    queue.add(info);
}

@Override
public void run() {
    // TODO Auto-generated method stub
    while(shouldRun)
    {   
        try {
            String info = queue.take();
            Callable<String> worker = new WorkerThread(info);
            System.out.println("submitting to pooler: " + info);
            pool.submit(worker);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }   
    }
}

/**
 * Inner class of proxy is a worker thread blocks until the pool has transactions complete as soon as they
 * are complete it will send them to server for completion.
 * @author Steve
 *
 */
class HttpSchedulerWorker  implements Runnable{

    public void run() {
        // TODO Auto-generated method stub
        while(true)
        {
            String vulnInfo = null;
            try {
                Future<String>  tmp = pool.take();
            //  Future<VulnInfo> tmp = pool.poll();
                if(tmp != null)
                    vulnInfo = tmp.get();
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } catch (ExecutionException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

            if(vulnInfo != null)
            {
                System.out.println("info was taken from pool completed: "  + vulnInfo);
            }



        }
    }

}

}

WorkerClass: this is the thread worker added to the executor pool and returns but in some cases never gets notified in the ThreadlockDemos ExecutorCompletionService pool?

import java.util.concurrent.Callable;

public class WorkerThread implements Callable<String>{


String info;
WorkerThread(String info)
{
    this.info = info;
}

//@Override
public String call() throws Exception {
    System.out.println("sending vuln info: " + info);
    return info;
}


}

Here is my test class simply adds items to queue. Here is a print out from my console of one that looks to have failed. Its adds to the queue works on it and returns the value. But the take() is never called any ideas why? It works sometimes and sometimes fails making it very hard for me to see what is wrong.I'd love to say its bug in java but I looked around did not see any issues with theses classes?

 public class HttpSchedulerThreadedUnitTest {

ThreadDeadlockDemo scheduler; 
public HttpSchedulerThreadedUnitTest(){

    setupScheduler();
    for(int i=0; i < 5;i++)
    {
        scheduler.add(i+"");
    }
}

private void setupScheduler()
{
    scheduler = new ThreadDeadlockDemo();
    scheduler.setThreadCount(1);
}

public static void main(String[] args)
{
    new HttpSchedulerThreadedUnitTest();
}

}

Console Print: this is it run never taking from the pool when the WorkerThread completes submitting to pooler: 0 submitting to pooler: 1 submitting to pooler: 2 sending vuln info: 0 submitting to pooler: 3 sending vuln info: 1 submitting to pooler: 4 sending vuln info: 2 sending vuln info: 3 sending vuln info: 4

Console Print : it actually doing taking itenms from pool returning! submitting to pooler: 0 submitting to pooler: 1 submitting to pooler: 2 submitting to pooler: 3 submitting to pooler: 4 sending vuln info: 0 info was taken from pool completed: 0 sending vuln info: 1 info was taken from pool completed: 1 sending vuln info: 2 info was taken from pool completed: 2 sending vuln info: 3 info was taken from pool completed: 3 sending vuln info: 4 info was taken from pool completed: 4

Answer 1

This is a lot of code. It would be nice if you can cut it down (by removing the http related parts etc). I'm also not sure what you mean by After that return the ExecutorCompletionService.take never sees it so never sees the blocking to return anymore items?

You could take a thread dump when it locks up and see which thread is locked in what point of the code.

Meanwhile, I do see some code that looks wrong.

while(requestQueue.isEmpty()){
            try {
                synchronized(this)
                {
                    wait();
                }
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            }

Here, you are synchronizing on a runnable object. This is almost always wrong because runnable objects are usually not accessed by more than one thread. Also you are testing for condition outside the synchronized statement. Usually you use wait as follows:

synchronized(lock){
    while(!condition){
        wait();
    }
}

However, I don't see any code that calls notify on the runnable. This may cause the program to hang. Basically you are waiting for something, but nobody is waking you up, so you wait indefinitely. Whether this is the cause of the problem you are facing, can be easily determined by looking at the thread dump when this happens.

If you are using a queue, best advice here would be to use a blocking queue for request queue. That way you don't have to do this wait/notify business altogether.