Spring Data JPA 垃圾收集

Question

I have a Spring Batch application with JpaPagingItemReader (i modified it a bit) and 4 Jpa repositories to enrich Model which comes from JpaPagingItemReader.

My flow is:

1. Select Model (page size = 8192 ), then i collect this List<Model> to Map<String, List<Model>> (group by id, because models not unique and i need to enrich by id) then enrich it with 4 custom JpaRepositories with native queries with IN clause, and merge them with Java 8 Streams .
2. Convert data to XML object and with Stax writing with MultiFileItemWriter to files, which are splitted no more than 20000 per file.

All works great, but today i tried to run flow with big amount of data from database. I generated 20 files ( 2.2 GB ). But sometimes i got OutOfMemory Java Heap (I had 1Gb XMS, XSS), then i up it to 2 GB and all works good, but in Instana i see, that Old gen Java memory is always 900 in use after GC. It is about 1.3-1.7Gb in use. So i start to think, how can i optimize GC of Spring Data Jpa objects. I think they are much time in memory. When i select Model with JpaPagingItemReader i detach every Model (with entityManager.detach ), but when i enrich Model with custom Spring Data Jpa requests i am not detaching results. Maybe the problem in this and i should detach them?

I do not need to insert data to database, i need just to read it. Or do i need to make page size less and select about 4000 per request?

I need to process 370 000 records from database and enrich them.

Answer 1

Solved. Added flags to my run configuration, and increase XMS and XMX twice.