Groovy Sql Paging Behavior

Question

I am using the groovy.sql.Sql class to query a database and process the results. My problem is that the ResultSet can be very large; so large that I risk running out of memory if I try to process the whole ResultSet at once. I know the Sql.rows() method supports paging using offset and max results parameters but I haven't been able to find a good example of how to use it (and I'm not certain that paging is what I'm looking for).

Basically, here's what I'm trying to do:

def endOfResultSet = false
for(int x = 1; !endOfResultSet; x+=1000){
    def result = sql.rows("Select * from table", x, 1000)
    processResult(result)
    endOfResultSet = result.size()!=1000
}

My question is if Groovy is smart enough to reuse the same result set for the sql.rows("Select * from table", x, 1000) call or if it will be repeatedly be running the same statement on the database and then paging to where the offset starts.

Your help is appreciated, Thanks!

Edit: What I'm trying to avoid is running the same query on the database multiple times. I'd like to run the query once, get the first 1,000 rows, process them, get the next 1,000 rows, etc... until all the rows are processed.

Answer 1

I assume you've seen this blog post about paging?

To answer your question, if we look at the code for the Sql class in Groovy, we can see that the code for rows(String,int,int) calls rows(String,int,int,null)

And the code for that is:

    AbstractQueryCommand command = createQueryCommand(sql);
    ResultSet rs = null;
    try {
        rs = command.execute();
        List<GroovyRowResult> result = asList(sql, rs, offset, maxRows, metaClosure);
        rs = null;
        return result;
    } finally {
        command.closeResources(rs);
    }

So as you can see, it gets the full ResultSet , then steps through this inside the asList method , filling a List<GroovyRowResult> object with just the results you requested.

Edit (after the question was edited)

As I said in my comment below, I think you're going to need to write your own paging query for the specific database you are using... For example, with MySQL , your above query can be changed to:

def result = sql.rows( "SELECT * FROM table LIMIT ${Sql.expand x}, 1000" )

Other databases will have different methods for this sort of thing...I don't believe there is a standard implementation

Answer 2

Answer from above is not correct. If you dig deeper, you'll find that if the ResultSet is not TYPE_FORWARD_ONLY, then the "absolute" method of the ResultSet is invoked to position a server side cursor. Then maxRows are returned. If the ResultSet is TYPE_FORWARD_ONLY, then ResultSet.next() is invoked offset number of times, then maxRows are returned. The exact performance characteristics will depend on the underlying jdbc driver implementation, but usually you want a scrollable result set when using the paging feature.

The resultset is not reused between invocations. Sounds like you want something like streaming, not paging.

Also, I wrote the patch, btw.

http://jira.codehaus.org/browse/GROOVY-4622