How to run the liquibase docker image in Jenkins containerized docker environment

Question

I'm doing POC with liquibase docker image,

I would like to run the liquibase docker image in docker with Jenkins kubernetes POD template. unfortunately unable to make it.

And also I have attached the Jenkins file and my observation.

Jenkins File

def workspace_dir = "/home/jenkins/agent/workspace/${env.JOB_BASE_NAME}"
def project_name = "master-chart"
def isDeployerJob = (env.JOB_BASE_NAME).contains("deploy") ? "true" : "false"

// These variables come from the build parameters in the Jenkins job
def git_branch = git_branch
def release_version


if (isDeployerJob == "true") {
    // Extracting the release version from the branch
    def temp = git_branch.split("/")
    release_version = temp[temp.length - 1]

    switch(environment) {
      case "dev":
        hs_jdbc_url="jdbc:postgresql://40.xx.xx.xx:5432/dbname"
        db_username="username"
        db_password="pwd"
        break
      default:
        break
    }
} 

pipeline {

  agent {
    kubernetes {
      cloud 'eks-tools-13'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: azcli-kubectl-helm
    image: internal.docker.cioxhealth.com/azcli-kubectl-helm
    command:
      - cat
    tty: true
  - name: docker
    image: docker
    command:
      - cat
    tty: true
    privileged: true
    volumeMounts:
      - name: dockersock
        mountPath: /var/run/docker.sock
  volumes:
    - name: dockersock
      hostPath:
        path: /var/run/docker.sock
"""
    }
  }

  stages {
        
    stage('Install Database Scripts') {
      when {
        expression {
          "${isDeployerJob}" == "true"
        }
      }
      steps {
        container('docker') { 
            sh """
               docker run --rm --network="host" -v ${workspace_dir}/db:/liquibase/changelog liquibase/liquibase --url=${hs_jdbc_url} --changeLogFile=db.changelog-master.yaml --driver=org.postgresql.Driver --username=${db_username} --password=${db_password} --logLevel=info  update
            """
        }
      }
    }   
  
  }
}

For verifying the files, I have getting into running container

Jenkins Master Node: ls -ltr /home/jenkins/agent/workspace/master-chart-deploy/db

total 4
drwxr-xr-x    3 1000     1000            21 Nov  6 04:35 sql
drwxr-xr-x    3 1000     1000            21 Nov  6 04:35 rollback
drwxr-xr-x    4 1000     1000            35 Nov  6 04:35 migration
-rw-r--r--    1 1000     1000           154 Nov  6 04:35 db-master-changelog.yaml
drwxr-xr-x    2 1000     1000            38 Nov  6 04:35 changelog

Docker container on master-chart-deploy-259-qxrn5-nqq7j-hhlb8

ls -ltr /home/jenkins/agent/workspace/master-chart-deploy/db

total 4
drwxr-xr-x    3 1000     1000            21 Nov  6 04:35 sql
drwxr-xr-x    3 1000     1000            21 Nov  6 04:35 rollback
drwxr-xr-x    4 1000     1000            35 Nov  6 04:35 migration
-rw-r--r--    1 1000     1000           154 Nov  6 04:35 db-master-changelog.yaml
drwxr-xr-x    2 1000     1000            38 Nov  6 04:35 changelog

Liquibase Container

docker run --rm '--network=host' -v /home/jenkins/agent/workspace/master-chart-deploy/db:/liquibase/changelog liquibase/liquibase -- ls -ltr /liquibase/changelog

total 0

Files are not available in the liquibase running container. due this the following error has been occurred.

Error:

Starting Liquibase at 14:50:38 (**version 4.1.1** #10 built at 2020-10-12 19:24+0000)
[2020-11-05 14:50:38] INFO [liquibase.lockservice] Successfully acquired change log lock
[2020-11-05 14:50:38] INFO [liquibase.lockservice] Successfully released change log lock
Unexpected error running Liquibase: db-master-changelog.yaml does not exist
For more information, please use the --logLevel flag
[2020-11-05 14:50:38] SEVERE [liquibase.integration] Unexpected error running Liquibase: db-master-changelog.yaml does not exist
liquibase.exception.ChangeLogParseException: db-master-changelog.yaml does not exist
    at liquibase.parser.core.yaml.YamlChangeLogParser.parse(YamlChangeLogParser.java:27)
    at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:337)
    at liquibase.Liquibase.lambda$update$1(Liquibase.java:229)
    at liquibase.Scope.lambda$child$0(Scope.java:160)
    at liquibase.Scope.child(Scope.java:169)
    at liquibase.Scope.child(Scope.java:159)
    at liquibase.Scope.child(Scope.java:138)
    at liquibase.Liquibase.runInScope(Liquibase.java:2277)
    at liquibase.Liquibase.update(Liquibase.java:215)
    at liquibase.Liquibase.update(Liquibase.java:201)
    at liquibase.integration.commandline.Main.doMigration(Main.java:1760)
    at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:361)
    at liquibase.Scope.lambda$child$0(Scope.java:160)

• May I know, What did I do wrong in this case? and why files are not available in liquibase running container?
• Is this a problem, because of file permissions due to Docker in Docker case?
• Is there any other way I can achieve this?

Thank you in advance for the help.

Answer 1

I think you are somehow messing with docker configuration. From documentation it looks like liquibase expects that you mount everything inside /liquibase/changelog directory.

And in your command you are mapping your changelogs to /app/liquibase :

docker run --rm --network="host" -v ${workspace_dir}/db:/app/liquibase liquibase/liquibase --url=${hs_jdbc_url} --changeLogFile=db.changelog-master.yaml --classpath=/app/liquibase --driver=org.postgresql.Driver --username=${db_username} --password=${db_password} --logLevel=info  update

so instead of that I'd use this:

docker run --rm --network="host" -v ${workspace_dir}/db:/liquibase/changelog liquibase/liquibase --url=${hs_jdbc_url} --changeLogFile=db.changelog-master.yaml  --driver=org.postgresql.Driver --username=${db_username} --password=${db_password} --logLevel=info  update

note: I've removed --classpath=/app/liquibase if you rely on it, if you've added some additional driver or something else you should probably include it again, but try to read about it at first. I think the documentation is pretty good.

Answer 2

you must specify full path when you use docker run in jekins pipeline:

--changeLogFile=/app/liquibase/db.changelog-master.yaml 

Define in jenkins pipeline :

environment {
        HOME = '.'
    }

Answer 3

Change log file is the main point from where Liquibase looks for configuration. If we do not define change log file path in Spring Boot, it considers db/changelog/db.changelog-master.yaml as default path for YAML format. As we will go with XML format, we need to set spring.liquibase.change-log=classpath:/db/changelog/changelog-master.xml for change log file path in application.properties file. You can set logging level of liquibase logs by setting log level in logging.level.liquibase property. Other properties in given below properties file are for H2 database configuration.