Parallel processing Windows batch file

Question

I have a pretty big number of files that need to be converted to a different format. The converting is done via a Java-JAR-File that gets takes the filename as a parameter. I now have a Windows batchfile that uses a for loop to loop through all the files (there is a file that contains a list of all files that need to be converted)

for /F %%i in (all_files.txt) do call java -cp %Classpath% de.xyz.Convert -xml %%i .\xml

Now the machine I want to do this on has eight cores. The number of files is about 360.000 and I would like it to take as little time as possible, so I'd like to use as many cores as possible. How would I go about using multiple cores as easy as possible? Is Windows going to be doing that on its own?

Answer 1

Ok, because I hadn't actually done it before, I knocked this up. It's not great, and the lib I used was a jar I created to crash after 2 mins.. Hopefully you'll be able to reverse engineer this for your needs.

package test;

import java.io.IOException;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;


public class Test {
    public static void main(String[] args) throws InterruptedException, IOException {
        BlockingQueue<Runnable> runnableQueue = new LinkedBlockingQueue<>();
        ExecutorService executorServ = new ThreadPoolExecutor(8, 8, 1, TimeUnit.MINUTES, runnableQueue);
        runnableQueue.add(new RunCrash("Example")); // Add one for each file...
        executorServ.shutdown();
        while(!executorServ.isTerminated()) {
            // running
        }
    }
}

class RunCrash implements Runnable {

    private String fileName;
    RunCrash(String fileName) {
        this.fileName = fileName;
    }

    @Override
    public void run() {
        System.out.println(fileName);
        try {
            crash.CrashMe.main(new String[]{fileName});
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

Oh, you can let the main thread die before the others finish, I believe the JVM will keep the executor and associated queue. :)