春季批处理的批处理作业

Question

I have following job to be processed at certain time interval or on ad-hoc basis.

Steps in the job are:

1. Call twitter api and gather tweets for bunch of users and write them in a file

2. Read them from a file and Process them

3. Dump processed results in database

I want UI as well, where I can trigger a job on ad-hoc basis, moreover I should be able to provide parameters to it from UI.

Was thinking of Spring batch for this but it is for more of a read -> process -> write kind of job. Here in first step I am generating data which is read by second step. Not sure if I can still use Spring batch for this OR there could be better way for this.

Answer 1

Using spring batch, you can design your job as steps, each step has it's own reader, processor, writer.

<job id="yourJobID" >
    <step id="gatherTweet" next="processTweet">
        <tasklet>
            <chunk reader="tweetCollector" writer="tweetFileWriter"/>
        </tasklet>
    </step>
    <step id="processTweet">
        <tasklet>
            <chunk reader="tweetFileWriterReader" processor="tweetProcessor" writer="tweetDataBaseWriter"/>
        </tasklet>
    </step> 
</job>

For the UI to launch the job you can use spring batch admin.

Answer 2

Alternatively you could configure your job and steps using Java rather than using XML.

For example myJob bean is defined below and uses step0, step1 and step2 (the steps are just beans too):

// Job Definition //
@Bean
public Job myJob(JobBuilderFactory jobs)
{
    return jobs.get("My Twitter Job")
            .incrementer(new RunIdIncrementer())               
            .from(step0()).next(step1()).next(step2())
            .end()
            .build();
}

...

// Custom ItemReader that is Autowired in //    
@Autowired
protected ItemReader<TweetDto> gatherTweetItemReader;

...

@Bean
public Step step0()
{
    return steps.get("Step0 - gatherTweet")
            .tasklet(gatherTweetItemReader)
            .allowStartIfComplete(true) // Always run this step //
            .build();
}

// ... Step1 and Step2 definitions ... //