如何在单个 JDBC 连接中执行多个 JPA 查询

Question

In a Spring-boot application, I am using JPA with Hibernate.

I have a custom select query method with parameterised IN Clause defined in JPA Repository. Since the the data is huge, I am executing multiple times the same query with small batches of ID passed as parameters. But I am not able to Improve the performance.

There are around 200000 records and the Time taken is 1 min. Which is very Huge.

Following are the things that i tired

1. Selecting only the required columns, instead of the entire POJO -> did reduce the query time by 8 seconds

2. When i analysed the Hibernate statics: the following are the logs found:

   ( 382016470 nanoseconds spent acquiring 1 JDBC connections; 0 nanoseconds spent releasing 0 JDBC connections; 798909958 nanoseconds spent preparing 1 JDBC statements; 305523944 nanoseconds spent executing 1 JDBC statements; 0 nanoseconds spent executing 0 JDBC batches; 0 nanoseconds spent performing 0 L2C puts; 0 nanoseconds spent performing 0 L2C hits; 0 nanoseconds spent performing 0 L2C misses; 0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections); 0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)

It takes time to create JDBC Connection even with Hikari connection Pool. Following are the hikari properties

 spring.datasource.hikari.connection-timeout=120000
    spring.datasource.hikari.maximum-pool-size=100
    spring.datasource.hikari.minimum-idle=30
 spring.jpa.hibernate.connection.provider_class=org.hibernate.hikaricp.internal.HikariCPConnectionProvider
    spring.datasource.hikari.pool-name=HikariConnectionPool
    spring.datasource.hikari.data-source-properties.cachePrepStmts=true
    spring.datasource.hikari.data-source-properties.useServerPrepStmts=true

3. And For each jpa query JDBC connection is established, prepared. Even after setting the following properties

   spring.datasource.hikari.data-source-properties.cachePrepStmts=true spring.datasource.hikari.data-source-properties.useServerPrepStmts=true

I have kept the number of parameters for IN Clause constant

Code Snippnet

//JPA Reposistry

public interface CarRepository extends CrudRepository<Car, String>, JpaSpecificationExecutor<Car> {

@Query(SELECT u.carName as carName,
    u.carId as
    carId from
    Car u
    where u.carId in (:carIds))
List<CarP> findCarProjections@Param("carIds")Set<String> carIds);

}

//Service
    @Component public carService

{
@Autowired
CarRepository carRepo;

public void processCars(List<carId> carIds)
{

List<List<String>> carIdPartioned = partionCarIds(carIds)
ExecutorService es = Executors.newFixedThreadPool(10);
for(List<String> carIdPart : carIdPartioned)
es.submit(new ProcessCarInThread(carIdPart,carRepo));

}

//Each Call Creates a new connection to JDBC 
//Takes time to prepare Statement ( as given in hibernate statics )
class ProcessCarInThread implements Runnable
{
List<String> carIds;
CarRepository carRepo;
public ProcessCarInThread(List<String> carIds, CarRepository carRepo )
{
this.carIds=carIds;
this.carRepo=carRepo;
}
       @Override
    public void run() {
//query 1
List<CarP> cars=carRepo.findCarProjections(carIds);
//query 2
List<SomeotherEntity> cars=otherRepo.findSomethingElse(params);
//even query 1 and query2 is not executing in a single JDBC connection
//Do Something
}

}

I want to improve the performance, any suggestions are welcome.

Any way to avoid JDBC query preparation time or JDBC connection acquiring time?

Answer 1

Very surprisingly, a similar performance issue was faced in my current application post PROD deployment. And fortunately we were able to fix it.

Root cause : we observed that using multiple input parameter in @Query is causing performance issue.

@Query(SELECT u.carName as carName, u.carId as carId from Car u where u.carId in (:carIds))

Solution : Use entityManager instead to form similar query dynamically and significant improvement is performance was observed