Need help on Java Resultset. Running out of Heap Space

Question

Need help in effective memory management for below scenario. I am fetching data from two different databases and comparing data in Java.(currently testing on single database with two queries).

As 9.8 million records needs to be compared, I am copying 50k records each time and loading into ArrayList and comparing using Binarysearch. Though I am clearing (assigning to null and running gc) the arraylist after every iteration, I am getting Heap space error(Assigned 1GB RAM) after comparing 2.5 million records.

Where is the memory leakage in my query?

Query1= select empno,ename from table1 order by empno;
Query2= select empno,ename from table2 order by empno;

ResultSet rs1 = st1.executeQuery(query1);
ResultSet rs2 = st2.executeQuery(query2);               
for (;;) {
    ArrayList<String> al = new ArrayList<String>();
    ArrayList<String> al1 = new ArrayList<String>();

    if (totalRecords1 == Ubound)
        break;

    Lbound = Ubound + 1;
    Ubound = min(Ubound + 50000, totalRecords1);
    System.out.println("Lbound : " + Lbound);
    System.out.println("Ubound : " + Ubound);

    for (int i = Lbound; i <= Ubound; i++) {
        recordConcat1 = ""; recordConcat2 = "";
        String recordConcat1 = "", recordConcat2 = "";
        rs1.next();
        rs2.next();

        recordConcat1 = recordConcat1 + rs1.getString(z) + " ǀ ";
        recordConcat2 = recordConcat2 + rs2.getString(z) + " ǀ ";

        al.add(recordConcat1);

        al1.add(recordConcat2);

    }  /* End of First Lap */ 

    System.out.println("End of Lap : "+lap++);

    int index =0;
    for(int like=0;like<al.size();like++) {

        if(Collections.binarySearch(al1,al.get(like))>=0)
            continue;
        else {
            System.out.println("Not matched : "+ al.get(like));
            break;
        }
    }

    al =null;
    al1=null;
    System.gc();

} /* End of Infinite Loop */ 

Answer 1

Instead of setting the ArrayLists to null and calling GC. Call ArrayList.clear() That will not release the memory of the ArrayLists but reuse it. Moreover, because you have a fixed upper bound at 50000, pass that number to the constructor of ArrayList to avoid dynamic memory reallocation while automatically growing the ArrayLists from their default initial size 10.

On the other side, if I read properly, what you are trying to achieve is tell whether every element of one table is at the other. Because the data is sorted you can do this without loading chunks into memory but reading both ResultSets in parallel.

I think that your algorithm will not work properly even without the out of memory error. You read 50K from table A and 50K from table B. Now search each element of chunk B on chunk A. But what if an element within the first 50000 of B is at position 50001 of A. You'll think it's not there but actually it is. Right?