Java, Multi Threading, ExecutorService

Question

Following is some parts of my code, which uses Threading. The purpose is to retrieve all the records from database (approx. 5,00,000) and send them alert email messages. The problem I am facing is the variable emailRecords becomes very heavy and too much time is taken to send email message. How can I make it fast by using multi-threading such that 5,00,000 records are processed parallelly? I tried to use ExecutorService but got confused in implementing it. I got mixed up in the method checkName(), getRecords() and sendAlert(). All these 3 methods are used relevantly. So, where to use executorService ??

Please provide me the suggestion how to proceed with the following code and which part needs editing? Thanks in advance!!

public class sampledaemon implements Runnable {

    private static List<String[]> emailRecords = new ArrayList<String[]>();

    public static void main(String[] args) {
        if (args.length != 1) {
            return;
        }

        countryName = args[0];

        try {
            Thread t = null;
            sampledaemon daemon = new sampledaemon();
            t = new Thread(daemon);
            t.start();
        } catch (Exception e) {
            e.printStackTrace()
        }

    }

    public void run() {
        Thread thisThread = Thread.currentThread();
        try {
            while (true) {
                checkName(countryName);
                Thread.sleep(TimeUnit.SECONDS.toMillis(10));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public void checkName(String countryName) throws Exception {
        Country country = CountryPojo.getDetails(countryName)

        if (country != null) {
            getRecords(countryconnection);
        }
    }

    private void getRecords(Country country, Connection con) {
        String users[] = null;
        while (rs.next()) {
            users = new String[2];
            users[0] = rs.getString("userid");
            users[1] = rs.getString("emailAddress");
            emailRecords.add(props);

            if (emailRecords.size() > 0) {
                sendAlert(date, con);
            }
        }
    }

    void sendAlert(String date, Connection con) {
        for (int k = 0; k < emailRecords.size(); k++) {
            //check the emailRecords and send email 
        }
    }
}

Answer 1

The advantage of using the FixedThreadPool is that you don't have to do the expensive process of creating the threads again and again , its done at the beginning ...see below..

ExecutorService executor = Executors.newFixedThreadPool(100);

ArrayList<String> arList =  Here your Email addresses from DB will go in ;

for(String s : arList){

     executor.execute(new EmailAlert(s));

 }



public class EmailAlert implements Runnable{

  String addr;

   public EmailAlert(String eAddr){


         this.addr = eAddr;

       }


  public void run(){


     // Do the process of sending the email here..

  }

 }

Answer 2

From what i can tell is that you would most likely be single threaded data retrieval, and multi-threaded for the e-mail sending. Roughly, you'd be cycling through your result set and building a list of records. When that list hits a certain size, you make a copy and send off that copy to be processed in a thread, and clear the original list. At the end of the result set, check to see if you have unprocessed records in your list, and send that to the pool as well.

Finally, wait for the threadpool to finish processing all records.

Something along these lines:

protected void processRecords(String countryName) {
  ThreadPoolExecutor executor = new ThreadPoolExecutor(10, 10, 10, TimeUnit.SECONDS, 
        new ArrayBlockingQueue<Runnable>(5), new ThreadPoolExecutor.CallerRunsPolicy());

   List<String[]> emaillist = new ArrayList<String>(1000);

   ResultSet rs = ....

   try {
     while (rs.next()) {
        String user[] = new String[2];
        users[0] = rs.getString("userid");
        users[1] = rs.getString("emailAddress");

        emaillist.add(user);
        if (emaillist.size() == 1000) {
            final List<String[]> elist = new ArrayList<String[]>(emaillist);
            executor.execute(new Runnable() {
                public void run() {
                    sendMail(elist);
                }
            }
            emaillist.clear();
        }
     }
   }
   finally {
     DbUtils.close(rs);
   }

   if (! emaillist.isEmpty()) {
            final List<String[]> elist = emaillist;
            executor.execute(new Runnable() {
                public void run() {
                    sendMail(elist);
                }
            }
            emaillist.clear();
   }

   // wait for all the e-mails to finish.
   while (! executor.isTerminated()) {
       executor.shutdown();
       executor.awaitTermination(10, TimeUnit.DAYS);
   }


}

Answer 3

Creating a second thread to do all of the work in instead of doing the same work in the main thread isn't going to help you avoid the problem of filling up the emailRecords list with 5 million records before processing any of them.

It sounds like your goal is to be able to read from the database and send email in parallel. Instead of worrying about the code, first think of an algorithm for the work you want to accomplish. Something like this:

1. In one thread, query for the records from the database, and for each result, add one job to an ExecutorService
2. That job sends email to one person/address/record.

or alternatively

1. Read records from the database in batches of N (50, 100, 1000, etc)
2. Submit each batch to the executorService