SQLite Queries Extremely Slow in Java

Question

So I am trying to use SQLite with a fairly basic SQL query (for those not familiar with GLOB, its similar to LIKE):

SELECT * FROM dictionary where word GLOB '[paple][paple][paple][paple][paple]';

I can run this in SQLite Manager and it takes around 50ms to retrieve all the records. Now I write the following code in Java and it takes almost 1.5 seconds which seems ridiculously slow in comparison. I understand that it might take a bit longer but 1450ms longer is unacceptably slow:

Connection conn = DriverManager.getConnection("jdbc:sqlite:dictionary.sqlite");
Statement stat = conn.createStatement();

long start = System.currentTimeMillis();
ResultSet rs = stat.executeQuery("SELECT * FROM dictionary where word GLOB '[paple][paple][paple][paple][paple]';");

while (rs.next()) {
    System.out.println("word = " + rs.getString("word"));
}

rs.close();
conn.close();

long end = System.currentTimeMillis();
System.out.println("Took: " + (end - start));

I have a feeling that every time I call ResultSet.next() it has to re-query the database since it doesn't get all the records right away but I'm not 100% sure. I feel like there should be a much more efficient way to do this. So my question is does anyone know how to improve the Java code to be much faster?

PS: I am using sqliteJDBC . Could the implementation here be slowing me down? Just a thought I had.

Answer 1

Every time you call ResultSet#getString(String) , you're forcing a lot of work to be done. See the JDBC driver 's code for its internal method RS#findColumn(String) . Note that it doesn't cache the column-name-to-column-ordinal-index mapping. For every row in the result set you inspect, you're suffering multiple string comparison and case conversion operations.

Try replacing your use of ResultSet#getString(String) with ResultSet#getString(int) . First, early on outside the while loop , figure out the index of the column you wish to extract. (Note that it would be much better to replace your SELECT * with an explicit column list, in which case you'd already know the ordinal index of each column.)

final int indexWord = rs.findColumn("word");

Then, during iteration, use the previously-determined index:

// Avoid concatenating:
System.out.print("word = ");
System.out.println(rs.getString(indexWord));

Let us know whether that optimization has a noticeable impact.

Answer 2

I was facing the same problem using a small database. My code was similar to this:

public LinkedList<Person> getByType(Type type) {
    LinkedList<Person> list = new LinkedList<>();
    String query = "SELECT * FROM person WHERE type_id = " + String.valueOf(type.getId());

    try {

        ResultSet rs = executeQuery(query); // Just calls statement.executeQuery(query);
        logTimestamp("After executeQuery");

        while (rs.next()) {
            logTimestamp("After rs.next");

            Person person = buildPersonFromResultSet(rs); // Just instances a new Person(rs.getLong("id"), rs.getString("name"));
            logTimestamp("After buildPersonFromResultSet");

            list.add(person);
            logTimestamp("After list.add");

            // Each loop iteration takes less than 1 ms
        }

        // list.size() is 26

        logTimestamp("After the last rs.next"); // After the last rs.next(), it was taking 4 seconds!
    } catch (Exception e) {
        LOGGER.error("Could not list. Query=[" + query + "]", e);
    }

    return list;
}

Through timestamped logs, I noticed that a slowdown of 4 seconds was happening only in the last call to the rs.next() method. I took a look at the SQLite JDBC driver source code ( https://bitbucket.org/xerial/sqlite-jdbc/src ) and saw that there are a lot of stuff happening when the "fetch" cursor finds out he is on the last row. I tried to increase the statement's fetch size (as pointed out in other answers), but no success. I heard that the database tables should be indexed to ease that job. When I checked my tables, I was surprised because there was no indexes in the primary and foreign keys. Some database tools don't create indexes by default, so I did it and now the last iteration takes less than 1ms too.

So, summing up:

My SQLite database had no indexes. After creating them for the primary and foreign keys, all the loop takes 20 ms instead of 4 seconds.

Answer 3

Quite old :) but we had exactly the same issue: a query that returns ~1500 results, executes in 50-100ms in the SQLite CLI, executes in 40'000 ms with the JDBC driver.

99% of the time was spent in rs.next

We upgraded the sqlite-jdbc library from 3.7 to latest (3.8.11) and the performance was roughly multiplied by 1000.

Answer 4

The Java code looks fine to me. The main issue is that it is going to do a linear table scan, which could be rather slow on a large database, and an index on the word column won't help (or at least won't help a lot).

What is the underlying version of SQLite that you're using? Using the current release might enable more optimizations. (I ask because sqliteJDBC is a few years old, yet SQLite is embedded into the driver — of course, since it's an embedded database and not a DB server — and there's been quite a few releases since then.)