liquibase无法更新数据库和/或速度很慢

Question

We face a very strange issue when we use Liquibase to execute the following changesets:

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd">

    <changeSet id="add companyinfo.identifier" author="domi">
        <addColumn tableName="companyinfo">
            <column name="identifier" type="VARCHAR(50)" />
        </addColumn>
    </changeSet>

    <changeSet id="add uk companyinfo.identifier" author="domi">
        <addUniqueConstraint    columnNames="identifier" 
                                tableName="companyinfo" 
                                constraintName="uk_companyinfo_identifier"/>
    </changeSet>

</databaseChangeLog>

The changesets add a new column ( identifier ) to a table companyinfo and define a unique constraint on the same column. The table has about 1'500'000 records.

These changeSets are executed against MySQL (version 5.5.54) with the org.liquibase:liquibase-maven-plugin:3.5.3:update maven plugin/goal triggered by a Jenkins pipeline script. (Jenkins and DB are located at the US east coast). The JDBC Driver we use is: mysql:mysql-connector-java:5.1.41

Usually we don't have any issues with liquibase, but when we run this changeset we always get an errors like this after exactly 2 hours:

[INFO] --- liquibase-maven-plugin:3.5.3:update (default) @ persistence ---
[INFO] ------------------------------------------------------------------------
[INFO] Executing on Database: jdbc:mysql://yyyy.xxxx.net:3306/my_db?characterEncoding=utf8
INFO 5/2/17 5:11 PM: liquibase: Successfully acquired change log lock
INFO 5/2/17 5:11 PM: liquibase: Reading from DATABASECHANGELOG
SEVERE 5/2/17 7:11 PM: liquibase: src/main/resources/db/db.changelog-master.xml: db/changelog/db.changelog-companyinfo_identifier.xml::add companyinfo.identifier::domi: Change Set db/changelog/db.changelog-companyinfo_identifier.xml::add add companyinfo.identifier::domi failed.  Error: Communications link failure

The last packet successfully received from the server was 7,200,055 milliseconds ago.  The last packet sent successfully to the server was 7,200,054 milliseconds ago. [Failed SQL: ALTER TABLE companyinfo ADD identifier VARCHAR(50) NULL]
INFO 5/2/17 7:11 PM: liquibase: db/changelog/db.changelog-companyinfo_identifier.xml::add companyinfo.identifier::domi: Successfully released change log lock
SEVERE 5/2/17 7:11 PM: liquibase: db/changelog/db.changelog-companyinfo_identifier.xml::add companyinfo.identifier::domi: Could not release lock
liquibase.exception.LockException: liquibase.exception.DatabaseException: Error executing SQL UPDATE DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1: No operations allowed after connection closed.
    at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:283)
    at liquibase.Liquibase.update(Liquibase.java:218)
    at liquibase.Liquibase.update(Liquibase.java:192)
    at liquibase.Liquibase.update(Liquibase.java:335)
    at org.liquibase.maven.plugins.LiquibaseUpdate.doUpdate(LiquibaseUpdate.java:33)
    at org.liquibase.maven.plugins.AbstractLiquibaseUpdateMojo.performLiquibaseTask(AbstractLiquibaseUpdateMojo.java:30)
    at org.liquibase.maven.plugins.AbstractLiquibaseMojo.execute(AbstractLiquibaseMojo.java:394)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: liquibase.exception.DatabaseException: Error executing SQL UPDATE DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1: No operations allowed after connection closed.
    at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:68)
    at liquibase.executor.jvm.JdbcExecutor.update(JdbcExecutor.java:231)
    at liquibase.executor.jvm.JdbcExecutor.update(JdbcExecutor.java:205)
    at liquibase.lockservice.StandardLockService.releaseLock(StandardLockService.java:267)
    ... 28 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
    at com.mysql.jdbc.Util.getInstance(Util.java:408)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
    at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1198)
    at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1193)
    at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2388)
    at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2372)
    at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:52)
    ... 31 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 7,200,055 milliseconds ago.  The last packet sent successfully to the server was 7,200,054 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)

When I run the same commands as Liquibase executes from my laptop, not via JDBC but from within the SequelPro MySql client connected to the same DB as above (Client in Switzerland, DB at US east coast), it successfully finishes without any issues in a about 15 minutes.

These are the statements Liquibase executes:

--  *********************************************************************
--  Update Database Script
--  *********************************************************************
--  Change Log: src/main/resources/db/db.changelog-master.xml
--  Ran at: 5/3/17 7:55 AM
--  Against: XXXX@CCCCC@jdbc:mysql://yyyy.xxxx.net:3306/my_db?characterEncoding=utf8
--  Liquibase version: 3.5.3
--  *********************************************************************

--  Lock Database
UPDATE DATABASECHANGELOGLOCK SET LOCKED = 1, LOCKEDBY = 'xxxxx (192.168.1.24)', LOCKGRANTED = '2017-05-03 07:55:44.564' WHERE ID = 1 AND LOCKED = 0;

--  Changeset db/changelog/db.changelog-companyinfo_identifier.xml::add companyinfo.identifier::domi
ALTER TABLE companyinfo ADD identifier VARCHAR(50) NULL;

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('add companyinfo.identifier', 'domi', 'db/changelog/db.changelog-companyinfo_identifier.xml', NOW(), 838, '7:b2d3082917bf3ff3aecb6cbc363a5e9c', 'add companyinfo.identifier', '', 'EXECUTED', NULL, NULL, '3.5.3', '3790945685');

--  Changeset db/changelog/db.changelog-companyinfo_identifier.xml::add uk companyinfo.identifier::domi
ALTER TABLE companyinfo ADD CONSTRAINT uk_companyinfo_identifier UNIQUE (identifier);

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('add uk companyinfo.identifier', 'domi', 'db/changelog/db.changelog-companyinfo_identifier.xml', NOW(), 839, '7:5d98affa45f814b9ad32bc9c954ed32b', 'addUniqueConstraint constraintName=uk_companyinfo_identifier, tableName=companyinfo', '', 'EXECUTED', NULL, NULL, '3.5.3', '3790945685');

--  Release Database Lock
UPDATE DATABASECHANGELOGLOCK SET LOCKED = 0, LOCKEDBY = NULL, LOCKGRANTED = NULL WHERE ID = 1;

From what I know, the DB has no timeout set that would match the 2 hour limit:

• wait_timeout is set to 28.8K seconds (8 hours)
• interactive_timeout : 28800
• salve_net_timeout : 3600 (1 hour)
• connect_timeout : 10
• lock_wait_timeout : 31536000
• net_read_timeout : 30
• net_write_timeout : 60
• innodb_lock_wait_timeout : 50

I think there are two questions which are interesting:

• what is causing liquibase to fail after two hours?
• why is the DDL execution so much slower with liquibase?

...but maybe both questions have the same answer

---

update: I have executed the exact same maven/liquibase command that I trigger from Jenkins now from my local env (also connecting to the same remote database in question) - and guess what: it works like a charm, no issue at all - the command finishes in about 20min!

mvn -f pom.xml process-resources -Pupdate-db -Dliquibase.username=xxx -Dliquibase.password=xxxx -Dliquibase.url=jdbc:mysql://yyyy.xxxx.net:3306/my_db:3306/yooture_ci?characterEncoding=utf8

So my last guess is that there must be something between our CI infrastructure and the DB that is causing the issue, but what? It works always, just not in this case...

Answer 1

We finally found the issue...

The problem is caused by the default AWS NAT configuration. The AWS documentation says:

If a connection that's using a NAT gateway is idle for 5 minutes or more, the connection times out

It's really low and it definitely breaks the default configuration of most OSes out there.