save（）上的Spring Boot JPARepository性能

Question

I have an issue where my spring boot application performance is very slow when inserting data.

I am extracting a large subset of data from one database and inserting the data into another database.

The following is my entity.

@Entity
@Table(name = "element")
public class VXMLElementHistorical {

@Id
@Column(name = "elementid")   
private long elementid;

@Column(name = "elementname")
private String elementname; 

Getter/Setter methods...    

I have configured a JPA repository

public interface ElementRepository extends JpaRepository<Element, Long> {

}

and call the save() method with my object

@Transactional 
public void processData(List<sElement> hostElements) 
throws DataAccessException { 

List<Element> elements = new ArrayList<Element>();    

for (int i = 0; i < hostElements.size(); i++) {
        Element element = new Element();
        element.setElementid(hostElements.get(i).getElementid());
        element.setElementname(hostElements.get(i).getElementname());
        elements.add(element);
    }

   try{
   elementRepository.save(elements);{
   //catch etc...

}

What is happening is that for each item, it is taking between 6 and 12 seconds to perform an insert. I have turned on hibernate trace logging and statistics and what is happening when I call the save function is that hibernate performs two queries, a select and an insert. The select query is taking 99% of the overall time.

I have ran the select query direct on the database and the result returns in nanoseconds. Which leads me to believe it is not an indexing issue however I am no DBA.

I have created a load test in my dev environment, and with similar load sizes, the over all process time is no where near as long as in my prod environment.

Any suggestions?

Answer 1

As @M. Deinum said in comment you can improve by calling flush() and clear() after a certain number of inserts like below.

int i = 0;
for(Element element: elements) {
    dao.save(element);
    if(++i % 20 == 0) {
        dao.flushAndClear();
    }

}

Answer 2

Instead of creating a list of elements and saving those, save the individual elements. Every now an then do a flush and clear to prevent dirty checking to become a bottleneck.

@PersistenceContext
private EntityManager entityManager;

@Transactional 
public void processData(List<sElement> hostElements) 
throws DataAccessException {     

for (int i = 0; i < hostElements.size(); i++) {
        Element element = new Element();
        element.setElementid(hostElements.get(i).getElementid());
        element.setElementname(hostElements.get(i).getElementname());
        elementRepository.save(element)
        if ( (i % 50) == 0) {
            entityManager.flush();
            entityManager.clear();
        }
}
entityManager.flush(); // flush the last records.

You want to flush + clear every x elements (here it is 50 but you might want to find your own best number.

Now as you are using Spring Boot you also might want to add some additional properties. Like configuring the batch-size.

spring.jpa.properties.hibernate.jdbc.batch_size=50 

This will, if your JDBC driver supports it, convert 50 single insert statements into 1 large batch insert. Ie 50 inserts to 1 insert.

See also https://vladmihalcea.com/how-to-batch-insert-and-update-statements-with-hibernate/

Answer 3

Since loading the entities seems to be the bottleneck and you really just want to do inserts, ie you know the entities don't exist in the database you probably shouldn't use the standard save method of Spring Data JPA.

The reason is that it performs a merge which triggers Hibernate to load an entity that might already exist in the database.

Instead, add a custom method to your repository which does a persist on the entity manager. Since you are setting the Id in advance, make sure you have a version property so that Hibernate can determine that this indeed is a new entity.

This should make the select go away.

Other advice given in other answers is worth considering as a second step:

• enable batching.
• experiment with intermediate flushing and clearing of the session.
• saving one instance at a time without gathering them in a collection, since the call to merge or persist doesn't actually trigger writing to the database, but only the flushing does (this is a simplification, but it shall do for this context)