how to improve the performance of bulk transaction in spring mvc

Question

i am working with java spring mvc with JPA and hibernate. If i want to process 2000 transactions, can i process them in batches of 500 transactions using threads putting in consideration that the single transaction should be processed by the same thread?is there any other way to do this?

Answer 1

Hibernate doesn't enable batching by default. This means that it'll send a separate SQL statement for each insert.

To insert so massive dataset in once you should use batch insert. Here the hibernate properties: hibernate.jdbc.batch_size = 50 and hibernate.order_inserts = true

The first property tells Hibernate to collect inserts in batches of 50 (50 is only used here for as example..). The order_inserts property tells Hibernate to take the time to group inserts by entity, creating larger batches.

Then you should choose between explicit or implicit flushing.

1. Implicit: just set the batch size property and hibernate will do the rest.
2. Explicit flush and clear

for (int i = 0; i < 10; i++) {
        if (i > 0 && i % BATCH_SIZE == 0) {
            entityManager.flush();
            entityManager.clear();
        }
        A a = new A();
        entityManager.persist(a);   
}

So why using 2nd option?

When we persist an entity, Hibernate stores it in the persistence context. For example, if we persist 2,000 entities in one transaction, we'll end up having 2,000 entity instances in memory, possibly causing an OutOfMemory...

Other problem... as you said you're using MySQL. hibernate batch option not work with IdGenerator IDENTITY so you can't use batch insert with AUTO-INCREMENT feature of mysql. As sequence generation doesn't exist in MySQL you're stuck with TABLE generator. But table generator is less performant than identity one. To mention Vlad mihalcea (quote from his famous book High Performance Java Persistence )

When using MySQL and need lot of inserts It is a good idea to use JOOQ framework for that.