Spring Batch工作设计-多个读者

Question

I'm struggling with how to design a Spring Batch job. The overall goal is to retrieve ~20 million records and save them to a sql database.

I'm doing it in two parts. First I retrieve the 20 million ids of the records I want to retrieve and save those to a file (or DB). This is a relatively fast operation. Second, I loop through my file of Ids, taking batches of 2,000, and retrieve their related records from an external service. I then repeat this, 2,000 Ids at a time, until I've retrieved all of the records. For each batch of 2,000 records I retrieve, I save them to a database.

Some may be asking why I'm doing this in two steps. I eventual plan to make the second step run in parallel so that I can retrieve batches of 2,000 records in parallel and hopefully greatly speed-up the download. Having the Ids allows me to partition the job into batches. For now, let's not worry about parallelism and just focus on how to design a simpler sequential job.

Imagine I already have solved the first problem of saving all of the Ids locally. They are in a file, one Id per line. How do I design the steps for the second part?

Here's what I'm thinking…

Read 2,000 Ids using a flat file reader. I'll need an aggregator since I only want to do one query to my external service for each batch of 2K Ids. This is where i'm struggling. Do I nest a series of readers? Or can I do 'reading' in the processor or writer?

Essentially, my problem is that I want to read lines from a file, aggregate those lines, and then immediately do another 'read' to retrieve the respective records. I almost want to chain readers together.

Finally, once I've retrieved the records from the external service, I'll have a List of records. Which means when they arrive at the Writer, I'll have a list of lists. I want a list of objects so that I can use the JdbcItemWriter out of the box.

Thoughts? Hopefully that makes sense.

Andrew

Answer 1

This is a matter of design and is subjective, but based on the Spring Batch example I found (from SpringSource) and my personal experience, the pattern of doing addtional reading in the processor step is a good solution to this problem. You can also chain together multiple processors/readers in the 'processor' step. So, while the names don't exactly match, i find myself doing more and more 'reading' in my processors.

[http://docs.spring.io/spring-batch/trunk/reference/html/patterns.html#drivingQueryBasedItemReaders][1]

Answer 2

Given that you want to call your external service just once per chunk of 2.000 records, you 'll actually want to do this service call in an ItemWriter . That is the standard recommended way to do chunk-level processing.

You can create a custom ItemWriter<Long> implementation. It will receive the list of 2.000 IDs as input, and call the external service. The result from the external service should allow you to create a List<Item> . Your writer can then simply forward this List<Item> to your JdbcItemWriter<Item> delegate.