Apache Camel SQL component select all records in a batch mode

Question

I'm using apache camel as an ETL from ( select *... ) PostgreSQL to ( insert... ) MariaDB.

In the PostgreSQL there are a lot of records (more then 1M) and I want to do it in a batch way.

I've tried with several flag ( batchCount , batchSize ) but non of them worked.

I've also search in Apache Camel docs, without any success.

from("sql:SELECT * FROM my_schema.trees?dataSource=#postgersqlDataSource&batch=true")
            .convertBodyTo(String.class)
            .process(ex -> {
                log.info("batch insert for single table");
                List<Map<String, Object>> rows = ex.getIn().getBody(List.class);
                log.info(String.format("Value: %s", rows.size()));
            })
            .to("stream:out");

But the program crashed because it load everything to the memory (with 1 records it worked of course).

Any advise?

it runs overs Spring boot .

Answer 1

The batch option is only for producer (eg to). https://camel.apache.org/components/3.20.x/sql-component.html

Instead take a look at outputType=StreamList where you can combine this with split EIP (in streaming mode) to process the rows without loading all into memory.

This also mean you process 1 row at a time

from sql
  split
    process (1 row here)