并行任务批处理的设计方法

Question

I have a batch job that is expected to process around 1k task at a time. And each task roughly takes around 12 - 16 minutes on an average.

In current implementation , all tasks are pushed into a blocking queue. There is a thread that pops a task from this queue and processes it. For task we are using java's executor service for concurrent execution and once all of its sub tasks are processed we mark this task as complete and head to read another task from the queue. We cant optimize task processing time since it makes call to native library and are unaware of what it does internally.

With current implementation we are able to process around 300 task in more than 24 hrs.

I'm looking for appropriate platform or framework that could help to reduce the processing time.

I'm using Java 1.7,OSGI and Apache Karaf as container

PS : The task here is breaking down of certain images ranging from 500 MB - 4 GB into small chunks and storing it into jpeg format

Answer 1

For horizontal scaling I would use a messaging system. Simply put all the tasks into a JMS queue. Then start karaf on a cluster of machines and let each listen on the queue. JMS will then automatically feed the processes round robin. So the load will be distributed.