How can I fix my program with ExecutorService?

Question

So, I'm new in Java.

I wrote relatively simple program that does something with a lot of files.

It was slow, I wanted to run more threads than one. With little StackOverflow Community help I made something like this:

public class FileProcessor {
    public static void main(String[] args)
    {
        // run 5 threads
        ExecutorService executor = Executors.newFixedThreadPool(5);
        int i;

        // get first and last file ID to process
        int start = Integer.parseInt(args[0]);
        int end = Integer.parseInt(args[1]);

        for (i = start; i < end; i++)
        {
            final int finalId = i; // final necessary in anonymous class
            executor.submit(new Runnable() 
            {
                public void run() 
                {
                    processFile(finalId);
                }
            });
        }
    }

    public static void processFile(int id)
    {
        //doing work here
    }
}

This is really really simple multithreading solution and it does what I want. Now I want to fix/improve it because I guess I'm doing it wrong (program never ends, uses more memory than it should etc.).

1. Shall I reduce number of Runnable objects existing in memory at the same time? If I should - how can I do it?

2. How can I detect, that all job is done and exit program (and threads)?

Answer 1

If you want to reduce the number of threads running, just reduce the size you pass to the fixed thread pool constructor. As for termination, call shutdown and awaitTermination on the executor service. But that will only reduce the number of active threads, not the number of Runnables you're creating in your loop.

Answer 2

when you said program never ends, uses more memory than it should , it could be due to many reasons like,

1) processFile() might be doing some heavy I/O operations (or) it is blocked for some I/O data.

2) There could be a potential dead lock if there is any common data object share

Your thread logic itself, pretty straight forward with ThreadPoolExecutor and I believe the problem is with the code in processFile() .

Since you initialized the pool with 5, The ThreadPoolExecutor makes sure that there are only 5 active threads doing the work irrespective of how many possible threads you want to create.

So In this case, I would focus more on application logic optimization than thread management.

If you are really concerned about how many Runnable objects you want to create? Then That's a trade-off between your application requirement and available resources.

If each thread task is independent and there is a execution time limit for all of those threads, then you create more threads in pool and add more resources.

When you define a PoolExecutor with 5 threads at a time limit, and create 10,000 threads then obviously they have to wait as a Future Task in memory until a thread is available.