how to avoid “lock timeout” when updating DB using multiple threads?

Question

I am trying to update a table using multiple threads. But I am not updating the same records/rows at the same time. I am grouping the table into different groups and trying to update them simultaneously. However, I am getting the locked timeout error all the time.

I am using Hibernate, Spring MVC, ThreadPoolTaskExecutor and MySQL. I am getting the data from another DB schema and updating my own database. The data is huge which is why i want to use multi threads so it can be done faster. However, it's producing "lock timeout" error. Can anyone help please? thanks for your good heart.

I call sessionFactory.getCurrenSession() to update the database table.

here is my config:

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
        destroy-method="close" p:driverClassName="${jdbc.driverClassName}"
        p:url="${jdbc.url}" p:username="${jdbc.username}" p:password="${jdbc.password}">
    </bean>

    <bean id="sessionFactory"
        class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="configLocation">
            <value>classpath:hibernate.cfg.xml</value>
        </property>
        <property name="hibernateProperties">
            <props>
                <prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
                <prop key="hibernate.show_sql">true</prop>
            </props>
        </property>
    </bean>

    <bean id="transactionManager"
        class="org.springframework.orm.hibernate4.HibernateTransactionManager">
        <property name="sessionFactory" ref="sessionFactory" />
    </bean>

    <bean id="taskExecutor"
        class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
        <property name="corePoolSize" value="5" />
        <property name="maxPoolSize" value="10" />
        <property name="WaitForTasksToCompleteOnShutdown" value="true" />
    </bean>

here is my stacktrace:

WARN : org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 1205, SQLState: 41000
ERROR: org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Lock wait timeout exceeded; try restarting transaction
Exception in thread "taskExecutor-5" Exception in thread "taskExecutor-4" Exception in thread "taskExecutor-2" org.hibernate.exception.LockTimeoutException: could not execute statement
    at org.hibernate.dialect.MySQLDialect$1.convert(MySQLDialect.java:407)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136)
    at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:103)
    at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:413)
    at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:282)
    at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1289)
    at org.hibernate.internal.QueryImpl.executeUpdate(QueryImpl.java:116)
   org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
    at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)
    ... 25 more

Answer 1

A "lock wait timeout" can always happen (even with a large amount of inserts in one transaction) and there is no silver bullet to solve it. But I managed to get around it when I was trying to update half of all the records in one (relative small) table while the other half was being modified by another server.

• Review all SQL statements in the transaction.
  Use explain to make sure indexes are used where possible. Remove any statements that are not needed as part of the transaction.
• Optimize the order of the SQL statements in the transaction.
  This was a bit of trial and error for me, but try to imagine which order of SQL statements coming from multiple threads/connections might be easier to deal with for the database. In my case, just switching the order of two SQL statements made the "lock wait timeout" occur less frequent.
• Update smaller subsets.
  This finally solved the "lock wait timeout" for me. In my case there was an indexed column that allowed me to divide the larger update set into smaller subsets. So now one big update transaction was turned into about ten smaller update transactions. Keep in mind though that you need to be able to continue the smaller transactions after a crash (ie data must remain consistent in such a way that your application can redo the operation and have the same result).

Whether or not multiple threads will improve the throughput (updated rows per second) remains to be seen: it depends on the size of the update sets (network latency) and how efficiently MySQL can handle the locks for the table(s) to update the rows. You might only see a marginal improvement when using two threads/connections instead of one.

[Edit] Also watch out for database triggers/procedures: they can impact performance in a bad manner.

Answer 2

Maybe you could try to lower isolation level . If it helps you can dig more. It should speed up also execution in multi threaded environment.

If you are using annotations you can achieve this by @Transactional(isolation=Isolation.READ_UNCOMMITTED) on top of your transactional class.

Answer 3

This appears to be a timeout on the database side. I'd guess that the database is the limiting factor, so adding threads in your application doesn't help.

If you want to use threads to speed things up, I'd suggest using only two threads. While one thread reads from the other database, the second thread writes to the MySQL database.

Note that if both databases are on the same database server, even that won't help. You would need a faster database or a beefier database machine.