Spring : Fetch data from a table using multiple threads

Question

I have a table which has around 5,00,000 rows. I am willing to fetch data from this table using multithreading, in a batch of 50,000 each, as in each thread should 50,000 rows. Each thread's should be unique. I was able to create a thread:

    @Async
    public CompletableFuture<List<Employee>> findAllFromEmployee() {

        final List<Employee> employees = employeeRepository.findAll();
        return CompletableFuture.completedFuture(employees);
    }

I defined task executor like this:

@Bean(name = "taskExecutor")
    public Executor taskExecutor() {
        System.out.println("Creating Async Task Executor");
        final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(2);
        executor.setMaxPoolSize(2);
        executor.setQueueCapacity(100);
        executor.setThreadNamePrefix("EmployeeThread-");
        executor.initialize();
        return executor;
    }

I am not able understand how should I make sure that each threads reads only 50,000 fields. Thanks!!

Answer 1

Order your results from the table in a consistent manner (eg order by primary key). Assuming your primary key is a field called "id", you can use this:

employeeRepository.findAllOrderById()

Then from row 0 to 50,000, process them with Thread #1, 50,001 to 100,000 goes to Thread #2 and so on.