Data fetching using Spring Data JPA Stream

Question

Below is my controller

public class TestController {

@PersistenceContext
EntityManager entityManager;

@Autowired
ProductAltRepository productAltRepository;

@GetMapping("/findAll")
@Transactional(readOnly = true)
public void findAll() {

    Stream<ProductAltRelEntity> productAltRelEntities = productAltRepository.findAllProductAlts();
    List<ProductAltRelEntity> productAlts = Lists.newArrayList();
    productAltRelEntities.forEach(x -> {
        productAlts.add(x);
        entityManager.detach(x);
    });
    }

And here is the repository

@Repository
@Transactional
public interface ProductAltRepository
        extends JpaRepository<ProductAltRelEntity, Long>, JpaSpecificationExecutor<ProductAltRelEntity>{
@QueryHints(value = { @QueryHint(name = HINT_FETCH_SIZE, value = "" + Integer.MIN_VALUE),
            @QueryHint(name = HINT_CACHEABLE, value = "false"), @QueryHint(name = HINT_READONLY, value = "true"), })
    @Query("SELECT p FROM ProductAltRelEntity p")
    public Stream<ProductAltRelEntity> findAllProductAlts();
}

Requirement is to findAll(), pagination is giving OOM, So I thought of using Stream, but getting exception.

 [WARN] org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 17068, SQLState: 99999

2019-11-15 12:30:31.468 ERROR 7484 --- [nio-8082-exec-1] ohengine.jdbc.spi.SqlExceptionHelper: Invalid argument(s) in call: setFetchSize [ERROR] org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Invalid argument(s) in call: setFetchSize 2019-11-15 12:30:31.793 ERROR 7484 --- [nio-8082-exec-1] c.seddeGlobalExceptionHandler: Unhandled Exception: org.springframework.orm.jpa.JpaSystemException: could not execute query using scroll; nested exception is org.hibernate.exception.GenericJDBCException: could not execute query using scroll Caused by: org.hibernate.exception.GenericJDBCException: could not execute query using scroll Caused by: java.sql.SQLException: Invalid argument(s) in call: setFetchSize

Answer 1

For having a look to your source article in Medium.

I think the approach given in the article is much different than your's. You should at first open Stateless Session . I don't think that using the findAll() provided in spring-data uses that as it fetch all records from db.

Fix that would be a good start.

BUG POSSIBLE ROOT CAUSE:

I see that you are using @QueryHints your HINT_FETCH_SIZE is incorrect. It cannot work with Integer.MIN_VALUE as value as is equals -2^31 which is a negative value.

Answer 2

Reading the error log, you need to put the streaming code in a try block ie

try(Stream<ProductAltRelEntity> productAltRelEntities = productAltRepository.findAllProductAlts()){
    List<ProductAltRelEntity> productAlts = Lists.newArrayList();
    productAltRelEntities.forEach(x -> {
        productAlts.add(x);
        entityManager.detach(x);
    });
} catch(Exception ex) {ex.printStackTrace();}

Another thing I noticed is that you don't need to put @Transactional in ProductAltRepository since you already put it in TestController.

In order to stream the results we need to satisfy three conditions:

Forward-only resultset
Read-only statement
Fetch-size set to Integer.MIN_VALUE

Forward-only seems to be set already by Spring Data so we don't have to do anything special about that. Your code sample already has @Transactional(readOnly = true) annotation in the TestController which is enough to satisfy the second criteria, so @QueryHint(name = HINT_READONLY, value = "true") is not useful. The fetch-size seems useful.

Below is an extract from one of my projects:

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
import org.springframework.data.jpa.repository.Query;
import static org.hibernate.jpa.QueryHints.HINT_FETCH_SIZE;

public interface IssueRepo extends JpaRepository<Issue, Long>, JpaSpecificationExecutor<Issue> {    
    
    @QueryHints(value = @QueryHint(name = HINT_FETCH_SIZE, value = "" + Integer.MIN_VALUE))
        @Query("select t from Issue t ")
        Stream<Issue> getAll();
    }

Controller

public class ManageIssue {
@GetMapping("all/issues")
    @Transactional(readOnly = true)
    public String getAll() {
        System.out.println("Processing Streams ...");
        try(Stream<Issue> issuesList = issueRepo.getAll()){
        
        issuesList.forEach(issue -> {
             System.out.println(issue.getId()+" Issue "+issue.getTitle());
            });
        } catch(Exception ex) {ex.printStackTrace();}
        return "All";
    }
}

Answer 3

Let JPA Help you in pagination, why this much extra efforts?

 Page<ProductAltRelEntity> productAltRelEntity= productAltRepository
                    .findAll(PageRequest.of(page - 1, 10, Sort.by("createdon").descending()));