如何使用hiberate优化插入查询时间？

Question

I have about 1.4 million records of data but it is taking more than 3hr to insert them. I cannot seems to find the problem to it.

I read up and change from identity to sequence. It only improved by a little but it still take quite a long time to finish insert.

I am using:

• Hibernate 5
• Spring 4
• mssql 2014
• Wildfly 10

applicationContext-hibernate.xml

<tx:advice id="txAdvice">
        <!-- the transactional semantics... -->
        <tx:attributes>
            <tx:method name="*_TransNew" propagation="REQUIRES_NEW" />
            <tx:method name="*_NoTrans" propagation="NEVER" />

            <tx:method name="create*" propagation="REQUIRED" />
            <tx:method name="update*" propagation="REQUIRED" />
            <tx:method name="delete*" propagation="REQUIRED" />
            <tx:method name="add*" propagation="REQUIRED" />

            <tx:method name="generate*" propagation="REQUIRED" />
            <tx:method name="get*" propagation="REQUIRED" />
            <tx:method name="is*" propagation="REQUIRED" />

            <!-- other methods use the default transaction settings (see below) -->
            <tx:method name="*" read-only="true" />
        </tx:attributes>
    </tx:advice>


    <!-- ensure that the above transactional advice runs for any execution of 
        an operation defined by the following -->
    <aop:config>
        <aop:pointcut id="demoServiceOperations"
            expression="execution(* com.test.*.*.*(..))" />
        <aop:advisor advice-ref="txAdvice" pointcut-ref="demoServiceOperations" />
    </aop:config>

<bean id="sessionFactory"
        class="org.springframework.orm.hibernate5.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="hibernateProperties">
            <props>
                <prop key="hibernate.show_sql">true</prop>
                <prop key="hibernate.dialect">org.hibernate.dialect.SQLServer2012Dialect</prop>
                <prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory</prop>
                <prop key="hibernate.cache.use_second_level_cache">true</prop>
                <prop key="hibernate.cache.use_query_cache">true</prop>
                <prop key="hibernate.jdbc.batch_size">50</prop>
                <prop key="hibernate.order_inserts">true</prop>
                <prop key="hibernate.order_updates">true</prop>

                <prop key="hibernate.c3p0.min_size">5</prop>
                <prop key="hibernate.c3p0.max_size">20</prop>
                <prop key="hibernate.c3p0.timeout">1800</prop>
                <prop key="hibernate.c3p0.max_statements">50</prop>


            </props>
        </property>         
    </bean>

Umts.hbm.xml:

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<!-- Generated Nov 22, 2016 11:36:21 AM by Hibernate Tools 5.2.0.Beta1 -->
<hibernate-mapping>
    <class name="com.test.domain.Umts" table="TBLDM_UMTS" schema="dbo" catalog="DEMO" optimistic-lock="version" dynamic-update="true">
        <id name="umtsId" type="java.lang.Integer">
            <column name="UMTS_ID" />
            <generator class="org.hibernate.id.enhanced.SequenceStyleGenerator">
                <param name="optimizer">pooled-lo</param>
                <param name="increment_size">1</param>
                <param name="sequence_name">UMTS_SEQ</param>
            </generator>
        </id>
        <property name="cid" type="java.lang.Integer">
            <column name="CI" not-null="true" />
        </property>
        <property name="channelNo" type="java.lang.Integer">
            <column name="UARFCN" />
        </property>
        <property name="signalStrength" type="java.lang.Double">
            <column name="EC_IO" precision="53" scale="0" />
        </property>
        <property name="sc" type="java.lang.Integer">
            <column name="SC" />
        </property>
        <property name="latitude" type="java.lang.Double">
            <column name="LATITUDE" precision="53" scale="0" />
        </property>
        <property name="longitude" type="java.lang.Double">
            <column name="LONGITUDE" precision="53" scale="0" />
        </property>
         <property name="mcc" type="java.lang.Integer">
            <column name="MCC" not-null="true" />
        </property>
        <property name="mnc" type="java.lang.Integer">
            <column name="MNC" not-null="true" />
        </property>
        <property name="recvDate" type="date">
            <column name="RECV_DATE" length="10" />
        </property>
        <property name="recvTime" type="time">
            <column name="RECV_TIME" length="16" />
        </property>
    </class>
</hibernate-mapping>

Service class:

public void process(List<Umts> umtsList)
{
  for (int i = 0; i < umtsList.size(); i = i + PropertiesUtil.MAX_COMMIT_COUNT)
        {
            int min = i;
            int max = i + PropertiesUtil.MAX_COMMIT_COUNT;

            if (max > umtsList.size())
            {
                max = umtsList.size();
            }

            createUmts_TransNew(umtsList.subList(min, max));
        }
}
    @Override
        public void createUmts_TransNew(Collection list) 
        {
            // TODO Auto-generated method stub

            umtsDAO.saveAll(list);  
        }

DAO class:

@Transactional
    public void saveAll(Collection collection)
    {
        log.debug("** save all");
        try
        {
            if (collection != null && collection.size() > 0)
            {
                for (Object obj : collection)
                {
                    sessionFactory.getCurrentSession().saveOrUpdate(obj);                   
                }
            }
        }
        catch (RuntimeException re)
        {
            log.error("** save all failed", re);
            throw re;
        }
    }

** Edited Does connection pool plays a part here? Meaning does connection pool helps with the performance? Do i need to add the jar file to the wildfly 10 or to application itself?

Answer 1

First off, try increasing the pooled-lo value. Since you set it to 1, there is no optimisation/pooling going on - since every ID that needs to be fetched needs a call to the DB to get the actual value. If you have a bigger increment size, hibernate will pre-fetch/reserve a block of ID's to use for new entities, without a per-entity round-trip.

Not sure how the code you posted is executed, but I'm assuming you're inserting them sequentially in a single thread. You can:

• Use a thread-pool in which each thread takes a number of items from the list/queue to insert.
• The items inserted in a single transaction by one thread has ideally same size as the configured hibernate batch size to again, minimize roundtrips.
• Ensure the number of threads in the pool is in par with the connection pool size, so you don't end up blocking the worker threads when waiting for a connection.
• Ensure connection pool size is reasonable for the load your server can take and use a good connection pool (eg. HikariCP ). Here is an interesting writeup on connection pool size.