Spring Batch并行Tasklet

Question

I am currently building a spring batch application where several steps are executed. Except one, all the steps are simple tasklets (with no reader or writer) and they are responsible for various tasks like copying files, sending requests, starting batch (*.bat) files etc.

Most of the steps should be executed serially. In one specific step I want to start X *.bat files which could have a maximum of Y instances.

In an example, lets say I have 10 *.bat files, but I want to have maximum 5 parallel running. So the first 5 start together, when on of those finishes the next (6th) should start, till all 10 are processed.

Of course, when the execution of all 10 is finished, the next step should start (so it is a synchronous execution).

The questions:

1. is spring batch the correct way to go? (is it a step that should be 10 times executed with different parameters?)
2. or should I only execute the step once and develop a "thread-controller" that would allow 5 (or Y) maximum threads?

If 1 == true :) I guess I have to work with the taskExecutor, below I have an example, where I start the first step (lets say has to find out how many the X is), after that I have a flowParallel (that simply says, if there are more batches, start the step again), then I made a split to allow the parallel execution (currently only 3 steps, which of course I could add all X with a loop that are limited from "taskExecutor.setMaxPoolSize" , which I find stupid)

Flow flowInit = new  FlowBuilder<Flow>("flowInit")
            .from(stepS1)
            .end();


    Flow flowParallel = new  FlowBuilder<Flow>("flowParallel")
            .start(stepS1Parallel)
            .next(deciderOne)
            .on("thereAreMoreBatchesToExecute")
            .to(stepS1Parallel).end();


    final Flow splitFlow = new FlowBuilder<Flow>("splitFlow")
                    .start(flowParallel)
                    .split(new SimpleAsyncTaskExecutor())
                    .add(flowParallel, flowParallel, flowParallel)
                    .build();

    return jobs.get("dataLoadParallel")
            .start(flowInit)
            .next(splitFlow)
            .next(stepS1)
            .end().build();

So, what I am doing wrong? which way should I go?

Answer 1

If you want tu set a maximum concurrency, then you have to use the setConcurrencyLimit method of the SimpleAsyncTaskExecutor.

If you want to have several steps running in parallel, you need to instantiate unique steps and unique flows for every step. In your example above, you start the same instance of a step (stepS1Parallel) inside the same instance of a flow (flowParallel) in parallel. This means, that the same instance of a step is called with multiple threads and this will definitely screw things up.

So, you need to have a loop in which you create one instance of a step together with one instance of a flow for every *.bat file you want to process.

HTH