Which is the best way to go - store table data to java arraylist or access table when and where needed?

Question

The problem is like this:

I am retrieving latitude and longitude from a point and storing it into an agent class. The class has the members id, latitude and longitude and assosciated functions.

Now at the start of the java app, it reads all rows from a local MySQL table(having cordinate values) and stores it into an arraylist of these agent variables. It then reads through this arraylist and calls necessary functions and draws a graph at the end.

This works fine for a table of 8K - 10K rows but then for another table with about 200,000 rows, it starts giving Heap size error in java.

I am thinking of changing the code to access the db everytime coordinates of an agent is needed. This should reduce the memory required right? However, there are many loops using the Iterator of this agent ArrayList. I am still in a fix on how to handle that.

So which would be a better option - reading from the table everytime or proceed as it is and increase the java memory allocation? Thanks for reading.

Answer 1

try to only load data as you need it. We don't have very much information, but I am assuming there is no need for the user to see/interact with all 10,000 data points at once. Besides the physical limitations of the size of the device screen, realistically a user will just be overwhelmed with that much information.

I would try implementing a paging system. Again, I have no idea what you are using the Agent class for, but try to only load as many Agent s as you are showing to the user.

Answer 2

Your strategy has serious drawback: big amount of data is read from disk, then stored in database's memory, sent to java app and it finally placed in JVM's memory. This takes much time and resources.

Use cursors, then you 'll be able to read from db and iterate on resultset in java app at the same time and JVM won't have to allocate so much memory.

Some documentation:

• http://forge.mysql.com/wiki/Cursors
• http://dev.mysql.com/doc/refman/5.6/en/connector-j-reference-implementation-notes.html (Read ResultSet section)