Spring数据缓存启用批量提交？

Question

I have a spring-boot application, which imports a huge document containing several thousand entities (~50 different classes, all with the same super-class), to work on the persisted entities later on. The document contains a lot of references within the document (using rdf).

At the moment I persist every entity. If this entity (E1) has a reference to another one (E2) I look in the database if there is already an entity (E2) with this rdf-id and if not I first import this entity (E2) and then the first one (E1). (This can also have much longer reference-chains).

So I thought, that I have the following 2 bottlenecks: A lot of SELECTs and a lot of INSERTs. For the first one, I use the cache, which comes with spring (delete isn't realy needed for the import):

public interface IdentifiedObjectRepository<T extends IdentifiedObject> extends CrudRepository<T, Long>{
    @Cacheable(cacheNames = "cimCache", key = "#p0")
    public T findOneByRdfId(String rdfId);

    @CachePut(cacheNames = "cimCache", key = "#p0.rdfId")
    public <S extends T> S save(S entity);

    @CacheEvict(cacheNames = "cimCache", allEntries = true)
    public void delete(Long id);
}

Every class has a repository, which is a sub-interface of the above one.

As it turned out the SELECTs doesn't seem to be a bottleneck, unless I did something wrong in the code above, since the duration hasn't changed.

Now I need to address the second bottleneck. My idea was to work just on the cache and INSERT everything as a batch once the document is completely imported, but I don't know how to do this with the tool-set of spring. (I already had a solution, where I made my own cache with a lot of HashMaps ).

Answer 1

If I understand the problem correctly, I'd say you are using the wrong tool for the job or you a using the wrong approach.

Interrupting an import of one entity to import another one which it references sounds complicated and inefficient to me.

I would do one of the following.

Option 1: Use the database

1. Insert all the entities in the database (using Spring Data JPA or even just plain JDBC). But don't resolve the references yet, keep them just as Strings. Make sure you use batching.

2. Run a single update statement (per table) that sets all the references to the correct value.

Option 2: Do the same in memory

1. Load all entities, without resolving any references. Store them in a way that let's you easily and efficiently find a document by it's reference.

2. Resolve all the references.

3. Save it. Make sure batching is configured properly.