Spring Batch: Pass over names of read-in files to the next step

Question

I'm new to Spring Batch and I got started off using Spring Boot & Spring Batch official tutorial .

The problem I have is that, after reading in some files using spring batch, I want to store the names of those files and remove them in the next step.

I first tried StepExecutionContext to pass data but apparently the data that I wished to pass along was way to big. So I tried saving the name of the files that I read when I initialized my Reader and reading that file in my next step.

@Bean
public MultiResourceItemReader<Widget> widgetMultiReader() {
    MultiResourceItemReader<Widget> reader = new MultiResourceItemReader<Widget>();
    ClassLoader cl = this.getClass().getClassLoader();
    ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
    try {
        System.out.println("file:" + dataFileLocation + "Widget*.dat");
        Resource[] resources = resolver.getResources("file:" + dataFileLocation + "Widget*.dat");
        System.out.println("FOUND " + resources.length + " Widget.dat files");
        StringBuilder sb = new StringBuilder();
        for (int i=0; i<resources.length; i++) {
            System.out.println(resources[i].getFilename());
            sb.append(resources[i].getFilename());
            if(i < resources.length-1){
                sb.append("\n");
            }
        }
        reader.setResources(resources);
        reader.setDelegate(widgetReader());

        File tempFolder = new File(dataFileLocation + Constants.TEMP_FOLDER_NAME);
        tempFolder.mkdir();
        File tempFile = new File(tempFolder.getAbsolutePath(), "read_in_files.tmp");
        tempFile.createNewFile();
        FileWriter fileWriter = new FileWriter(tempFile, false);
        fileWriter.write(sb.toString());
        fileWriter.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
    return reader;
}

This was all working really well but turns out if I do this, the code inside the bean will execute automatically even if it's not defined in the job step. This apparently is because of dependency injection ( Spring boot spring.batch.job.enabled=false not able to recognize )

So I have to change my method yet again can you guys help me out here please?

All I want to do is to read in a set of text files, read in the data inside it, store it to the database (this is already done) THEN, remove/archive/move those files in the next step so if there's something like FlatFileParseException, entire job will stop and not remove the files.

Answer 1

Create a service that you can use to pass the data between the steps

By annotating the service with @JobScope or @StepScope it will only live during that step or job.

import org.springframework.batch.core.configuration.annotation.JobScope;
import org.springframework.stereotype.Service;

import java.io.File;
import java.util.List;

@Service
@JobScope
public class FilesService {

    private List<File> files;

    public List<File> getFiles() {
        return files;
    }

    public void setFiles(List<File> files) {
        this.files = files;
    }
}