如何使用 JPA 和 Hibernate 的多个处理线程来持久化实体

Question

There is one job which is running in a single threaded environment. This means I have main method and the main thread is responsible for completing the job.

I am using Spring and Hibernate.

On the high level I'm doing the following steps:

1. fetching data from a MySQL database using JDBC ( looping resultset and doing point 2. and 3. )

2. populating a Model using the data received from point 1.

3. validation, calling service layer, dao layer and storing entity in oracle db.

This flow is using a for loop. So 1 by 1 data insertion is there.

Now I want to do it in using multi threading.

Approach :

1. one thread will fetch the data and populate model object and put it in the queue.

2. multiple threads will dequeue object from queue and start point 3.

Can you help me in implementing this model. How to code this type of multithreading framework.

Answer 1

You can do it like this:

1. Define an ExecutorService:

    ExecutorService executorService = Executors.newFixedThreadPool(10);

2. Each for iteration should simply submit the new object to be inserted

    final RecordDTO record = ...; executorService.execute(new Runnable() { public void run() { insertService.save(record); } });

3. The insertService will have a @Transactional save method to insert each record using one of those 10 worker threads.

4. The connection pool size should be greater or equal to the number of workers.

However, it's much more efficient to use JDBC batch updates . Therefore, instead of sending just one entity to the worker thread, you can send multiple entries so that they are all inserted in one batch.

Answer 2

you need a single-producer multiple-consumer thread-safe queue. Take a look at LMAX disruptor , that's the best suit for you.

LMAX Disruptor is a High Performance Inter-Thread Messaging Library and it's the default messaging system for Intra-worker communication in Apache Storm The underlying data structure is a lock-free ring buffer. To make it fast, it use a lot of tricks to reduce false sharing

follow theget started tutorial , there is a very simple example you could refer to.