[英]SQLite Queries Extremely Slow in Java
So I am trying to use SQLite with a fairly basic SQL query (for those not familiar with GLOB, its similar to LIKE): 所以我试图使用SQLite和一个相当基本的SQL查询(对于那些不熟悉GLOB的人,它类似于LIKE):
SELECT * FROM dictionary where word GLOB '[paple][paple][paple][paple][paple]';
I can run this in SQLite Manager and it takes around 50ms to retrieve all the records. 我可以在SQLite Manager中运行它,它需要大约50ms来检索所有记录。 Now I write the following code in Java and it takes almost 1.5 seconds which seems ridiculously slow in comparison. 现在我用Java编写以下代码,它需要将近1.5秒,相比之下看起来非常慢。 I understand that it might take a bit longer but 1450ms longer is unacceptably slow: 据我所知,它可能需要更长的时间,但1450毫秒更长是令人无法接受的慢:
Connection conn = DriverManager.getConnection("jdbc:sqlite:dictionary.sqlite");
Statement stat = conn.createStatement();
long start = System.currentTimeMillis();
ResultSet rs = stat.executeQuery("SELECT * FROM dictionary where word GLOB '[paple][paple][paple][paple][paple]';");
while (rs.next()) {
System.out.println("word = " + rs.getString("word"));
}
rs.close();
conn.close();
long end = System.currentTimeMillis();
System.out.println("Took: " + (end - start));
I have a feeling that every time I call ResultSet.next() it has to re-query the database since it doesn't get all the records right away but I'm not 100% sure. 我有一种感觉,每当我调用ResultSet.next()时,它必须重新查询数据库,因为它没有立即获得所有记录,但我不是100%肯定。 I feel like there should be a much more efficient way to do this. 我觉得应该有一个更有效的方法来做到这一点。 So my question is does anyone know how to improve the Java code to be much faster? 所以我的问题是,是否有人知道如何更快地改进Java代码?
PS: I am using sqliteJDBC . PS:我正在使用sqliteJDBC 。 Could the implementation here be slowing me down? 这里的实施能否减缓我的速度? Just a thought I had. 只是我的想法。
Every time you call ResultSet#getString(String)
, you're forcing a lot of work to be done. 每次调用ResultSet#getString(String)
,都会强制执行大量工作。 See the JDBC driver 's code for its internal method RS#findColumn(String)
. 请参阅JDBC驱动程序的代码,了解其内部方法RS#findColumn(String)
。 Note that it doesn't cache the column-name-to-column-ordinal-index mapping. 请注意,它不会缓存column-name-to-column-ordinal-index映射。 For every row in the result set you inspect, you're suffering multiple string comparison and case conversion operations. 对于您检查的结果集中的每一行,您将遭受多个字符串比较和大小写转换操作。
Try replacing your use of ResultSet#getString(String)
with ResultSet#getString(int)
. 尝试使用ResultSet#getString(int)
替换ResultSet#getString(String)
使用。 First, early on outside the while
loop , figure out the index of the column you wish to extract. 首先, 在while
循环之外的早期,找出要提取的列的索引。 (Note that it would be much better to replace your SELECT *
with an explicit column list, in which case you'd already know the ordinal index of each column.) (请注意,用明确的列列表替换SELECT *
会好得多,在这种情况下,您已经知道每列的序数索引。)
final int indexWord = rs.findColumn("word");
Then, during iteration, use the previously-determined index: 然后,在迭代期间,使用先前确定的索引:
// Avoid concatenating:
System.out.print("word = ");
System.out.println(rs.getString(indexWord));
Let us know whether that optimization has a noticeable impact. 让我们知道优化是否会产生明显的影响。
I was facing the same problem using a small database. 我使用小型数据库遇到了同样的问题。 My code was similar to this: 我的代码与此类似:
public LinkedList<Person> getByType(Type type) {
LinkedList<Person> list = new LinkedList<>();
String query = "SELECT * FROM person WHERE type_id = " + String.valueOf(type.getId());
try {
ResultSet rs = executeQuery(query); // Just calls statement.executeQuery(query);
logTimestamp("After executeQuery");
while (rs.next()) {
logTimestamp("After rs.next");
Person person = buildPersonFromResultSet(rs); // Just instances a new Person(rs.getLong("id"), rs.getString("name"));
logTimestamp("After buildPersonFromResultSet");
list.add(person);
logTimestamp("After list.add");
// Each loop iteration takes less than 1 ms
}
// list.size() is 26
logTimestamp("After the last rs.next"); // After the last rs.next(), it was taking 4 seconds!
} catch (Exception e) {
LOGGER.error("Could not list. Query=[" + query + "]", e);
}
return list;
}
Through timestamped logs, I noticed that a slowdown of 4 seconds was happening only in the last call to the rs.next()
method. 通过带时间戳的日志,我注意到仅在最后一次调用rs.next()
方法时发生了4秒的减速。 I took a look at the SQLite JDBC driver source code ( https://bitbucket.org/xerial/sqlite-jdbc/src ) and saw that there are a lot of stuff happening when the "fetch" cursor finds out he is on the last row. 我看了一下SQLite JDBC驱动程序源代码( https://bitbucket.org/xerial/sqlite-jdbc/src ),看到当“fetch”光标发现他在的时候发生了很多事情。最后一排。 I tried to increase the statement's fetch size (as pointed out in other answers), but no success. 我试图增加语句的获取大小(正如其他答案所指出的那样),但没有成功。 I heard that the database tables should be indexed to ease that job. 我听说应该将数据库表编入索引以简化该工作。 When I checked my tables, I was surprised because there was no indexes in the primary and foreign keys. 当我检查我的表时,我很惊讶,因为主键和外键中没有索引。 Some database tools don't create indexes by default, so I did it and now the last iteration takes less than 1ms too. 默认情况下,某些数据库工具不会创建索引,所以我这样做了,现在最后一次迭代也需要不到1毫秒。
So, summing up: 所以,总结一下:
My SQLite database had no indexes. 我的SQLite数据库没有索引。 After creating them for the primary and foreign keys, all the loop takes 20 ms instead of 4 seconds. 在为主键和外键创建它们之后,所有循环都需要20 ms而不是4秒。
Quite old :) but we had exactly the same issue: a query that returns ~1500 results, executes in 50-100ms in the SQLite CLI, executes in 40'000 ms with the JDBC driver. 相当老:)但我们有完全相同的问题:一个返回~1500结果的查询,在SQLite CLI中执行50-100ms,使用JDBC驱动程序在40'000 ms内执行。
99% of the time was spent in rs.next 99%的时间花在rs.next上
We upgraded the sqlite-jdbc library from 3.7 to latest (3.8.11) and the performance was roughly multiplied by 1000. 我们将sqlite-jdbc库从3.7升级到最新版(3.8.11),性能大致乘以1000。
The Java code looks fine to me. Java代码对我来说很好。 The main issue is that it is going to do a linear table scan, which could be rather slow on a large database, and an index on the word
column won't help (or at least won't help a lot). 主要问题是它将进行线性表扫描,这在大型数据库上可能相当慢,并且word
列上的索引将无济于事(或者至少不会有太大帮助)。
What is the underlying version of SQLite that you're using? 您正在使用的SQLite的基础版本是什么? Using the current release might enable more optimizations. 使用当前版本可能会启用更多优化。 (I ask because sqliteJDBC is a few years old, yet SQLite is embedded into the driver — of course, since it's an embedded database and not a DB server — and there's been quite a few releases since then.) (我问,因为sqliteJDBC已经存在了几年,但SQLite已嵌入到驱动程序中 - 当然,因为它是一个嵌入式数据库而不是数据库服务器 - 从那时起就有不少版本。)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.