简体繁体 English

Java处理大量数据

[英]Java handling large amounts of data

原文 2010-01-29 00:58:57 4 4 java/ database/ optimization

I have a Java application that needs to display large amounts of data (on the order of 1 million data points). 我有一个Java应用程序，它需要显示大量数据（大约一百万个数据点）。 The data doesn't all need to be displayed at the same time but rather only when requested by a user. 不需要同时显示所有数据，而仅在用户请求时才显示。 The app is a desktop app that is not running with an app server or hitting any centralized database. 该应用程序是一个桌面应用程序，未与应用程序服务器一起运行或未访问任何集中式数据库。

My thought was to run a database on the machine and load the data in there. 我的想法是在计算机上运行数据库并在其中加载数据。 The DB will be read only most of the time, so I should be able to index to help optimize queries. 在大多数时候，数据库都是只读的，因此我应该能够建立索引以帮助优化查询。 If I'm running on a local system, I'm not sure if I should try and implement some caching (I'm not sure how fast the queries will run, I'm currently working on them). 如果我在本地系统上运行，则不确定是否应该尝试实现一些缓存（我不确定查询将以多快的速度运行，目前正在使用它们）。

Is this is a logical way to approach the problem or would there be better approaches? 这是解决问题的逻辑方法，还是会有更好的方法？

Thanks, Jeff 谢谢杰夫

4 个解决方案

Display and data are two different things. 显示和数据是两个不同的东西。

You don't give any details about either, but it could be possible to generate the display in the background, bringing in the data one slice at a time, and then displaying when it's ready. 您没有提供任何详细信息，但是可以在后台生成显示，一次将数据引入一个切片，然后在准备好时显示。 Lots of anything could cause memory issues, so you'll need to be careful. 很多东西都可能导致内存问题，因此您需要小心。 The database will help persist things, but it won't help you get ten pounds of data into your five pound memory bag. 数据库将帮助持久化事物，但并不能帮助您将十磅的数据放入五磅的存储袋中。

UPDATE: If individuals are only reading a few points at a time, and display isn't an issue, then I'd say that any database will be able to handle it if you index the table appropriately. 更新：如果个人一次只读几点，并且显示不是问题，那么我想说，如果您对表进行适当索引，那么任何数据库都将能够处理它。 One million rows isn't a lot for a capable database. 对于功能强大的数据库来说，一百万行并不多。

Embedded DB seems reasonable. 嵌入式DB似乎很合理。 Check out JavaDB / Derby or H2 or HSQLDB . 查看JavaDB / Derby或H2或HSQLDB 。

Sqlite with a java wrapper is fine too. 带有Java包装器的 Sqlite也很好。

It really depends on your data. 这实际上取决于您的数据。 Do multiple instances request the data? 是否有多个实例请求数据？ If not, it is definitely worth to look for a simple SQLite database as the storage. 如果没有，那么绝对值得寻找一个简单的SQLite数据库作为存储。 It is just a single file on your file system. 它只是文件系统上的一个文件。 No need to set up a server. 无需设置服务器。

Well, depends on data size. 好吧，取决于数据大小。 1 Million integers for example isnt that much, but 1 Million data structures/classes or whatever with, lets say, 1000 Bytes size is much. 例如，一百万个整数并没有那么多，但是一百万个数据结构/类或具有1000字节大小的任何东西都可以说是很多。

For small data: keep them in memory For large data: i think using the DB would be good. 对于小数据：将它们保留在内存中对于大数据：我认为使用DB会很好。

Just my opinion :) 只是我的观点：）

edit: 编辑：

Of course it depends also on the speed you want to achieve. 当然，这还取决于您要达到的速度。 If you really need high speed and the data is big you could also cache some of them in memory and leave the rest in the db. 如果您确实需要高速且数据量很大，则还可以将其中一些缓存在内存中，并将其余的保留在数据库中。