Best way to read huge db tables in java

Question

I have a database with a table which has millions of records. I am using spring jpa to access database. Now when i try using findAll() on that table i am getting memory error. The size of that table is around 150mb and i have given 1gb heap to jvm. My question is why it is consumng so much heap memory. If this isn't the correct approach should i go with the pagination followed by calling garbage collector.

Answer 1

Never get all records in a single shot, get all values in chunks Ie pages and then you will get rid of outofmemory issue. Spring-data-* modules helps you to do it effectively. You can pass page as parameter and retrieve data.

One sample to achieve is findAll method which takes pagable as parameter and returns you page.

Answer 2

Spring data modules provide a simple (and a bit naive) way to handle big result set using pagination.
If your requirement is to show only a chunk of the result set per time, then Spring pagination is enough.

If you need to show the entire result set (tipically a log table), than you can:

• Use jpa vendor features like CursoredStream (EclipseLink)
• Use JDBC ResultSet

This features let you execute a query, and use the result like an Iterator (more or less).
The jvm memory will be loaded little by little during the result set scrolling, based on the fetchSize you defined.
With this method you can discard the records already sent to client, mantaining only the latest in memory.