Wait for specific condition to become true in threads

Question

I have an application that makes HTTP requests to a site, ant then retrives the responses, inspects them and if the contain specific keywords, writes both the HTTP request and response to an XML file. This application uses a spider to map out all the URLS of a site and then sends request(each URL in the sitemap is fed to a separate thread that sends the request). This way I wont be able to know when all the requests have been sent. At the end of all I request i want to convert the XML file to some other format. So in order to find out when the request have ended I use the following strategy :

I store the time of each request in a varible (when a new request is sent at a time later than the time in the variable, the varible is updated). Also I start a thread to monitor this time, and if the difference in the current time and the time in the varible is more than 1 min, I know that the sending of requests has ceased. I use the following code for this purpose :

class monitorReq implements Runnable{
    Thread t;
    monitorReq(){
        t=new Thread(this);
        t.start();
    }
    public void run(){
        while((new Date().getTime()-last_request.getTime()<60000)){
             try{ 
                 Thread.sleep(30000);//Sleep for 30 secs before checking again
             }
             catch(IOException e){
                 e.printStackTrace(); 
             }
        }
        System.out.println("Last request happened 1 min ago at : "+last_request.toString());
        //call method for conversion of file
    }
}

Is this approach correct? Or is there a better way in which I can implement the same thing.

Answer 1

Your current approach is not reliable. You will get into race conditions - if the thread is updating the time & the other thread is reading it at the same time. Also it will be difficult to do the processing of requests in multiple threads. You are assuming that task finishes in 60 seconds..

The following are better approaches.

If you know the number of requests you are going to make before hand you can use a CountDownLatch

main() {
   int noOfRequests = ..;
   final CountDownLatch doneSignal = new  CountDownLatch(noOfRequests);

   // spawn threads or use an executor service to perform the downloads
   for(int i = 0;i<noOfRequests;i++) {
      new Thread(new Runnable() {
         public void run() {
            // perform the download
            doneSignal.countDown();
         }
      }).start();
   }

   doneSignal.await();  // This will block till all threads are done.
}

If you don't know the number of requests before hand then you can use the executorService to perform the downloads / processing using a thread pool

main() {

  ExecutorService executor = Executors.newCachedThreadPool();
  while(moreRequests) {
    executor.execute(new Runnable() {
      public void run() {
        // perform processing
      }
    });
  }

  // finished submitting all requests for processing. Wait for completion
  executor.shutDown();
  executor.awaitTermination(Long.MAX_VALUE, TimeUnit.Seconds);

}

Answer 2

General notes:

1. classes in Java should start with Capital Letters

2. there seems to be no synchronization between your threads; access to last_request should probably be synchronized

3. Using System.currentTimeMillis() would save you some objects' creation overhead

4. swallowing an exception like this is not a good practice

Answer:

Your way of doing it is acceptable. There is not much busy waiting and the idea is as simple as it gets. Which is good.

I would consider changing the wait time to a lower value; there is so little data, that even doing this loop every second will not take too much processing power, and will certainly improve the rection time from you app.