JPA query: what is a proper way to select by a large set of possible values?

Question

What I have is a large (like hundreds of thousands) list of IDs. I want to get some values from the objects with this IDs with a JPA query.

Option 1 - using IN clause

List<Long> ids = /* ... */
entityManager.createQuery(
        "SELECT v.lat, v.lon FROM Vehicle v WHERE v.id IN (:ids)",
        Long[].class)
    .setParameter("ids", ids)
    .getResultList();

I'm afraid there might be a limit on IN clause contents.

Option 2 - query each ID separately

List<Long> ids = /* ... */
for (Long id : ids) {
    entityManager.createQuery(
            "SELECT v.lat, v.lon FROM Vehicle v WHERE v.id = :id",
            Long[].class)
        .setParameter("id", id)
        .getSingleResult();
}

That looks too slow as seems to be doing ids.size() requests.

What is a proper way to do this type of thing?

Answer 1

Neither of your methods have much merit. As mentioned in Grzegorz's comment, all databases have a limit on how many list items you can use. Also, there are limits on how many characters your query can contain. So lists are out.

Hundreds of thousands of individual queries seems (remember the posting guidelines Dan, be courteous) unwise.

If this were my problem, I'd look at how that list of id's got generated in the first place. It was probably a database query. If so, I would make a new version of that query to get the lats and lons. It might be a table join or it might be a subquery. Without knowing your database structure, I can't give you any detailed suggestions.

Answer 2

First thing that comes to mind:

1. Cut the id list in parts of say 1000 ids. A manageable size.
2. Run an IN query for each part.
3. Aggregate the result into a list.

For big data sets, the performance can be improved dramatically by doing (2) in parallel.