spring 批处理使用 spring 启动：从配置或命令行读取 arguments 并在作业中使用它们

Question

I am pretty new to spring technology. I am trying to build an ETL like app using spring batch with spring boot.

Able to run the basic job (read->process->write). Now, I want to read the arguments (like date, file name, type, etc) from a config file (later) or command line (can work with it now) and use them in my job.

Entry point:

// Imports
@SpringBootApplication
@EnableBatchProcessing
public class EtlSpringBatchApplication {

    public static void main(String[] args) {
        SpringApplication.run(EtlSpringBatchApplication.class, args);
    }

}

My batch configuration

// BatchConfig.java
// Imports
    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Autowired
    public MyDao myDao;

    @Bean
    public Job job() {
        return jobBuilderFactory
                .get("job")
                .incrementer(new RunIdIncrementer())
                .listener(new Listener(myDao))
                .flow(step1())
                .end()
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1").<myModel, myModel>chunk(1000)
                .reader(Reader.reader("my_file_20200520.txt"))
                .processor(new Processor())
                .writer(new Writer(myDao))
                .build();
    }

I have basic steps steps.

Reader.java has method to read flat file.

public static FlatFileItemReader<MyModel> reader(String path) {......}

Processor.java has process method defined. I added a @BeforeStep to fetch some details from DB required for processing.

public class Processor implements ItemProcessor<MyModel, MyModel> {

    private static final Logger log = LoggerFactory.getLogger(Processor.class);
    private Long id = null;

    @BeforeStep
    public void getId(StepExecution stepExecution) {
        this.id = stepExecution.getJobExecution().getExecutionContext().getLong("Id");
    }

    @Override
    public MyModel process(MyModel myModel) throws Exception {
    }
}

Writer.java is implementing ItemWriter and write code.

Listener.java extends JobExecutionListenerSupport and has overridden methods afterJob and beforeJob. Basically tried to use executioncontext here in beforeJob.

@Override
public void beforeJob(JobExecution jobExecution) {
    log.info("Getting the id..");
    this.id = myDao.getLatestId();
    log.info("id retrieved is: " + this.id);
    jobExecution.getExecutionContext().putLong("Id", this.id);
}

Now, what I am looking for is:

• The reader should get the file name from job arguments. ie when run the job, I should be able to give some arguments, one of them is file path.
• Later some methods (like get id, etc) require few more variables which can be passed as arguments to job ie run_date, type, etc.

In short I am looking for a way to,

• Pass job arguments to my app (run_date, type, file path etc)
• Use them in reader and other places (Listener, Writer)

Can someone provide me what addiitons I should do in my BatchConfig.java and other places, to read the job parameters (from command line or config file, whichever is easy)?

Answer 1

You can read the value of the of the job parameters set from the config file inside the reader or other classes within the spring batch execution context. Below is a snippet for reference,

application.yml file can have the below config,

batch.configs.filePath: c:\test

You can add the filePath read from the config to your job parameters when you start the job. Snippet of the class,

// Job and Job Launcher related autowires..

@Value("${batch.configs.filePath}")
private String filePath;

// inside a method block,
JobParameters jobParameters = new JobParametersBuilder().addLong("JobID", System.currentTimeMillis())
            .addString("filePath", filePath).toJobParameters();

try {
    jobLauncher.run(batchJob, jobParameters);
} catch (Exception e) {
    logger.error("Exception while running a batch job {}", e.getMessage());
}

One of the ways to access the Job Parameters is to implement StepExecutionListener to your reader Class to make use of its Overridden methods beforeStep and afterStep. Similar implementations can be performed to other classes as well,

public class Reader implements ItemReader<String>, StepExecutionListener {

private String filePath;

@Override
public void beforeStep(StepExecution stepExecution) {

    try {
        filePath = (String) stepExecution.getJobExecution().getExecutionContext()
                .get("filePath");
    } catch (Exception e) {
        logger.error("Exception while performing read {}", e);
    }
}


@Override
public String read() throws Exception {
    // filePath value read from the job execution can be used inside read use case impl

}

@Override
public ExitStatus afterStep(StepExecution stepExecution) {
    return ExitStatus.COMPLETED;
}

}

Answer 2

Both Spring Batch and Spring Boot reference documentation show how to pass parameters to a job:

• Running Jobs from the Command Line
• Running Spring Batch jobs from the Command Line

Moreover, Spring Batch docs explain in details and with code examples how to use those parameters in batch components (like reader, writer, etc):

• Late Binding of Job and Step Attributes