How to query inmemory objects by attribute?

Question

I'm getting a list of 300 json objects that are to be cached in memory for a day. During execution of the application I want to query objects by their properties.

Example:

@XmlRootElement(name = "persons")
@XmlAccessorType(XmlAccessType.FIELD)
class PersonsDTO {
    private List<PersonDTO> persons;

    public static class PersonDTO {
        private String name;
        private int age;
        //lots of more attributes
        private Address address;
    }
}

Here I'd like to run queries similar to a database, like:

findByname("john doe");
findByAgeBetween(10, 18);

Question: how could I best prepare the data for those "query" lookups? Create a HashMap for each query function where I then could just return the precalculated results?

Or is there any "database-like" inmemory system that I could use and that can be queried similar to a real database?

Answer 1

You could use a SQL based in memory database such as Derby or HSQLDB . You could use a NoSQL based in memory database, such as hazelcast or MapDB . You could use a caching solution that support queries like EHCache .

But 300 objects is tiny. You could just loop over them and have 'em in an array and only add Map based indexes if you really need to (that is doing a large number of queries on it).

It all comes down to:

1. How often are you going to query this dataset?
2. Are the queries fixed (I want Joe Blake), fixed query with parameters (I want everyone called X) or free-form?

If you are rarely going to query the dataset and the queries are fixed/fixed with parameters then I'd just loop over the dataset (Java 8 Streams API; the simplest solution). If you need to hit the dataset a lot but with fixed queries then I'd pre-calculate the results. If you need to hit the dataset a lot with a small number of fixed queries with parameters and I'd think about adding some home make indexes (hash map). If you need free-form queries I'd think about using an in-memory database (NoSQL or SQL) or the Java 8 Streams API.

Answer 2

I would just brute force it unless you have specific performance requirements. Scanning 300 entries should take less than 0.1 ms.

This would allow you to use the streams API built in.

private List<PersonDTO> persons;

public List<PersonDTO> findBy(Predicate<PersionDTO> test) {
    return persons.stream().filter(test).collect(Collectors.toList());
}


// findByName
List<PersonDTO> david = findBy(p -> p.getName().startsWith("David "));

// find by age
List<PersonDTO> youngAdult= findBy(p -> p.getAge() >= 18 && p.getAge() <= 30);

Answer 3

HSQLDB is an in-memory SQL database written in Java.

Otherwise you will need to design and build your own engine. You might have dedicated method names, like you show, and that avoids some of the query parsing aspects but you would still need to implement the queries themselves including however you define matches (exact, partial, case-sensitive, etc.).

Answer 4

300 is not a large number (although Spartans and Persians might argue with that...).

1. So in a simplistic solution you can just keep your objects in a list and just iterate through the list and peak the objects with needed properties. It is very inefficient but for 300 it will work just fine.

   2. A bit of a step up if you work with Java 8 you can use parallel streaming and filtering, that would utilize your processor resources in the best way. Still not terribly efficient.

   3. Yet another idea - add or generate a unique id for each object and create a mater map of your id to your DTO. Then create a separate maps for each property that you intend to search by. Each map should hold property value to the list of ids of DTOs with that value. Then you query your property map to find IDs and then search your master map to find DTOs. In general this is a classical trade off between performance and space.

Answer 5

In Java8 you could just use streams for this simple task. Of course you need to evaluate if it is good enough for your purposes, but this is a very quick and simple way to implement without needing setup a local database.

public List<PersonDTO> findByAgeBetween(min, max){
    List<PersonDTO> byAge = 
      personList.stream().
      filter(p -> p.age >= min).
      filter(p -> p.age <= max).
      collect(Collectors.toList());
    return byAge;
}

And similar for the other query