如何使用Spring Batch在並行模式下運行步驟

Question

我正在做春季批。 我有一個分區步驟（對象列表），然后是帶有Reader和Writer的從屬步驟。

我想以並行模式執行processStep 。 因此，我希望每個分區都有一個特定的Reader-Writer實例 。

目前，創建的分區使用相同的Reader-Writer實例。 因此，這些操作是在串行模式下完成的：讀取和寫入第一個分區，然后在第一個分區完成時對下一個分區進行相同的操作。

Spring Boot配置類：

@Configuration
@Import({ DataSourceConfiguration.class})
public class BatchConfiguration {

    private final static int COMMIT_INTERVAL = 1;

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

   @Autowired
   @Qualifier(value="mySqlDataSource")
   private DataSource mySqlDataSource;

   public static int GRID_SIZE = 3;

   public static List<Pojo> myList;

   @Bean
   public Job myJob() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {

      return jobBuilderFactory.get("myJob")
            .incrementer(new RunIdIncrementer())
            .start(partitioningStep())
            .build();
  }

  @Bean(name="partitionner")
  public MyPartitionner partitioner() {

    return new MyPartitionner();
  }

  @Bean
  public SimpleAsyncTaskExecutor taskExecutor() {

    SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
    taskExecutor.setConcurrencyLimit(GRID_SIZE);
    return taskExecutor;
  }

  @Bean
  public Step partitioningStep() throws NonTransientResourceException, Exception {

    return stepBuilderFactory.get("partitioningStep")
              .partitioner("processStep", partitioner())
              .step(processStep())
              .taskExecutor(taskExecutor())
              .build();
  }

  @Bean
  public Step processStep() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {

    return stepBuilderFactory.get("processStep")
            .<List<Pojo>, List<Pojo>> chunk(COMMIT_INTERVAL)
            .reader(processReader())
            .writer(processWriter())
            .taskExecutor(taskExecutor())
            .build();
  }

  @Bean
  public ProcessReader processReader() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {

    return new ProcessReader();
  }

  @Bean
  public ProcessWriter processWriter() {

    return new ProcessWriter();
  }
}

分區器類

public class MyPartitionner implements Partitioner{

@Autowired
private IService service;

@Override
public Map<String, ExecutionContext> partition(int gridSize) {

    // list of 300 object partitionned like bellow
    ...
    Map<String, ExecutionContext> partitionData = new HashMap<String, ExecutionContext>();

    ExecutionContext executionContext0 = new ExecutionContext();
    executionContext0.putString("from", Integer.toString(0));
    executionContext0.putString("to", Integer.toString(100));
    partitionData.put("Partition0", executionContext0);

    ExecutionContext executionContext1 = new ExecutionContext();
    executionContext1.putString("from", Integer.toString(101));
    executionContext1.putString("to", Integer.toString(200));
    partitionData.put("Partition1", executionContext1);

    ExecutionContext executionContext2 = new ExecutionContext();
    executionContext2.putString("from", Integer.toString(201));
    executionContext2.putString("to", Integer.toString(299));
    partitionData.put("Partition2", executionContext2);

    return partitionData;
 }
}

讀者班

    public class ProcessReader implements ItemReader<List<Pojo>>, ChunkListener {

    @Autowired
    private IService service;

    private StepExecution stepExecution;

    private static List<String> processedIntervals = new ArrayList<String>();

    @Override
    public List<Pojo> read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {

        System.out.println("Instance reference: "+this.toString());

        if(stepExecution.getExecutionContext().containsKey("from") && stepExecution.getExecutionContext().containsKey("to")){

            Integer from = Integer.valueOf(stepExecution.getExecutionContext().get("from").toString());
            Integer to = Integer.valueOf(stepExecution.getExecutionContext().get("to").toString());

            if(from != null && to != null && !processedIntervals.contains(from + "" + to) && to < BatchConfiguration.myList.size()){
                processedIntervals.add(String.valueOf(from + "" + to));
                return BatchConfiguration.myList.subList(from, to);
            }
        }

        return null;
    }

    @Override
    public void beforeChunk(ChunkContext context) {

        this.stepExecution = context.getStepContext().getStepExecution();
    }

    @Override
    public void afterChunk(ChunkContext context) { }

    @Override
    public void afterChunkError(ChunkContext context) { }

    }
  }

作家班

 public class ProcessWriter implements ItemWriter<List<Pojo>>{

    private final static Logger LOGGER = LoggerFactory.getLogger(ProcessWriter.class);

    @Autowired
    private IService service;

    @Override
    public void write(List<? extends List<Pojo>> pojos) throws Exception {

        if(!pojos.isEmpty()){
            for(Pojo item : pojos.get(0)){
                try {
                    service.remove(item.getId());
                } catch (Exception e) {
                    LOGGER.error("Error occured while removing the item [" + item.getId() + "]", e);
                }
            }
        }
    }
 }

您能告訴我代碼有什么問題嗎？

Answer 1

通過在我的讀寫器bean聲明中添加@StepScope來解決：

@Configuration
@Import({ DataSourceConfiguration.class})
public class BatchConfiguration {

   ...

   @Bean
   @StepScope
    public ProcessReader processReader() throws UnexpectedInputException, ParseException, NonTransientResourceException, Exception {

      return new ProcessReader();
   }

   @Bean
   @StepScope
   public ProcessWriter processWriter() {

     return new ProcessWriter();
   }

   ...

}

這樣，您為每個分區都擁有一個不同的塊（Reader-Writer）實例。