Trying to design a database in Java. So far I have a serializable object which I read and update when I need to. Good or bad idea?

Question

So, partly out of frustration with SQL syntax, I decided to try to implement my own Database. I don't need to perform very complex operations - only need to do row lookups and addition of new rows. I have two data structures, User and Circle . These are then put into Java's List and the final Database object looks like this (note that it implements Serializable ):

public class Database implements Serializable {
    private static final long serialVersionUID = 5790642843089065198L;

    List<User> users; 
    List<Circle> circles;

    public Database() {
        users = new ArrayList<User>();
        circles = new ArrayList<Circle>();
    }

}

Whenever I update the object, I also use ObjectOutputStream to "save" the object as a file. Whenever I read from the database, I user ObjectInputStream to "get" the object from a file. I also have a DatabaseHelper class which extends Thread . This class is rather long, but to put it simply, it initializes the Database object as a static variable. My question is not about a specific problem I am having, in fact I have confirmed that my code works as I expect it to work. The database is saved permanently when the program exits or even fails. I am also able to bring up a number of clients which all have an independent connection to the Database, but are also able to see each other's commits.

The problem which I am having has to do with design. Whenever I open up a Thread, the whole Database is read (it's updated only if a commit is made). How do enterprise databases work, say when you need to do a row lookup? Is that whole table read into memory from a file?

This may be a better question for cs.stackexchange.com, but any guidance is appreciated.

Answer 1

A common approach in databases is to use memory mapped files. This give you the convenience of having all the data in memory, almost immediately without having to wait for the data to actually load.

In Java this means mapping your files off heap and bring the data on heap as needed. Once you write the data off heap, it will be asynchronously saved by the OS.

I have a SharedHashMap which is designed for GC-free serialization, concurrent access across processes/threads and lazy persistence. Using memory mapped files means you can read a key/value by touching/reading only a very small number pages, the rest of the data doesn't need to be in memory.

Answer 2

Implementing a database on your own is not a good idea, as even elementar grouping/map-reduce features will cost you time you most certainly better want to spent for the business logic you're about to develop.

Java offers many possibilities for easy data access; certainly the most advanced are JPA (Java Persistence API; an official, generalized API for accessing nearly any database system without a need to write raw SQL queries), and Hibernate.

You may want to use one of these, as they implement exactly what you want (object serialization/hydration), are fast, reliable and use standard RDBMS in the background.