Inserting to and searching a large amount of data in Java

Question

I am writing a program in Java which tracks data about baseball cards. I am trying to decide how to store the data persistently. I have been leaning towards storing the data in an XML file, but I am unfamiliar with XML APIs. (I have read some online tutorials and started experimenting with the classes in the javax.xml hierarchy.)

The software has to major use cases: the user will be able to add cards and search for cards.

When the user adds a card, I would like to immediately commit the data to the persistant storage. Does the standard API allow me to insert data in a random-access way (or even appending might be okay).

When the user searches for cards (for example, by a player's name), I would like to load a list from the storage without necessarily loading the whole file.

My biggest concern is that I need to store data for a large number of unique cards (in the neighborhood of thousands, possibly more). I don't want to store a list of all the cards in memory while the program is open. I haven't run any tests, but I believe that I could easily hit memory constraints.

XML might not be the best solution. However, I want to make it as simple as possible to install, so I am trying to avoid a full-blown database with JDBC or any third-party libraries.

So I guess I'm asking if I'm heading in the right direction and if so, where can I look to learn more about using XML in the way I want. If not, does anyone have suggestions about what other types of storage I could use to accomplish this task?

Answer 1

While I would certainly not discourage the use of XML, it does have some draw backs in your context.

"Does the standard API allow me to insert data in a random-access way"

Yes, in memory. You will have to save the entire model back to file though.

"When the user searches for cards (for example, by a player's name), I would like to load a list from the storage without necessarily loading the whole file"

Unless you're expected multiple users to be reading/writing the file, I'd probably pull the entire file/model into memory at load and keep it there until you want to save (doing periodical writes the background is still a good idea)

I don't want to store a list of all the cards in memory while the program is open. I haven't run any tests, but I believe that I could easily hit memory constraints

That would be my concern to. However, you could use a SAX parser to read the file into a custom model. This would reduce the memory overhead (as DOM parsers can be a little greedy with memory)

"However, I want to make it as simple as possible to install, so I am trying to avoid a full-blown database with JDBC"

I'd do some more research in this area. I (personally) use H2 and HSQLDB a lot for storage of large amount of data. These are small, personal database systems that don't require any additional installation (a Jar file linked to the program) or special server/services.

They make it really easy to build complex searches across the datastore that you would otherwise need to create yourself.

If you were to use XML, I would probably do one of three things

1 - If you're going to maintain the XML document in memory, I'd get familiar with XPath ( simple tutorial & Java's API ) for searching.

2 - I'd create a "model" of the data using Objects to represent the various nodes, reading it in using a SAX. Writing may be a little more tricky.

3 - Use a simple SQL DB (and Object model) - it will simply the overall process (IMHO)

Additional

As if I hadn't dumped enough on you ;)

If you really want to XML (and again, I wouldn't discourage you from it), you might consider having a look a XML database style solution

Apache Xindice (apparently retired)

Or you could have a look at some other people think

• Use XML as database in Java
• Java: XML into a Database, whats the simplest way?

For example ;)