Using a Jenkins pipeline to checkout multiple git repos into same job

Question

I'm using the Jenkins Multiple SCM plugin to check out three git repositories into three sub directories in my Jenkins job. I then execute one set of commands to build a single set of artifacts with information and code drawn from all three repositories.

Multiple SCM is now depreciated, and the text recommends moving to pipelines. I tried, but I can't figure out how to make it work.

Here is the directory structure I'm interested in seeing from the top level of my Jenkins job directory:

$ ls
Combination
CombinationBuilder
CombinationResults

Each of those three sub-directories has a single git repo checked out. With the Multiple SCM, I used git, and then added the "checkout to a subdirectory" behavior. Here was my attempt with a pipeline script:

node('ATLAS && Linux') {
    sh('[ -e CalibrationResults ] || mkdir CalibrationResults')
    sh('cd CalibrationResults')
    git url: 'https://github.com/AtlasBID/CalibrationResults.git'
    sh('cd ..')
    sh('[ -e Combination ] || mkdir Combination')
    sh('cd Combination')
    git url: 'https://github.com/AtlasBID/Combination.git'
    sh('cd ..')
    sh('[ -e CombinationBuilder ] || mkdir CombinationBuilder')
    sh('cd CombinationBuilder')
    git url: 'https://github.com/AtlasBID/CombinationBuilder.git'
    sh 'cd ..'

    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

However, the git command seems to execute at the top level directory of the workspace (which makes some sense), and according to the syntax too, there doesn't seem to be the checkout-to-sub-directory behavior.

Answer 1

You can use the dir command to execute a pipeline step in a subdirectory:

node('ATLAS && Linux') {
    dir('CalibrationResults') {
        git url: 'https://github.com/AtlasBID/CalibrationResults.git'
    }
    dir('Combination') {
        git url: 'https://github.com/AtlasBID/Combination.git'
    }
    dir('CombinationBuilder') {
        git url: 'https://github.com/AtlasBID/CombinationBuilder.git'
    }

    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

Answer 2

You can checkout those three git repositories into three subdirectories by using the checkout SCM step three times like this:

stage('Checkout') {
 // Get CalibrationResults from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CalibrationResults']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/CalibrationResults.git']]
        ])
 // Get Combination from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'Combination']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/Combination.git']]
        ])
 // Get CombinationBuilder from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CombinationBuilder']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/CombinationBuilder.git']]
        ])

}

Answer 3

Here's Mine

    stage('CheckoutModule1') {
        steps {
            sh 'mkdir -p Module1'
            dir("Module1")
            {
                git branch: "develop",
                credentialsId: 'aaa',
                url: 'git@a.com:b/module1.git'
            }
        }
    }

    stage('CheckoutModule2') {
        steps {
            sh 'mkdir -p Module2'
            dir("Module2")
            {
                git branch: "develop",
                credentialsId: 'aaa',
                url: 'git@a.com:b/module2.git'
            }
        }
    }

Answer 4

If your repository has submodules use git checkout

pipeline {
agent {label 'master'}
stages{
    stage("Demo"){
        steps{

            echo 'Hello World'
        }
    }
    stage("Source"){
        parallel{
            stage('CalibrationResults'){
                steps{
                    echo 'Checking out CalibrationResults'
                    checkout([$class: 'GitSCM', branches: [[name: '*/CI-CD-Demo']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false, timeout: 60], [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: false, recursiveSubmodules: true, reference: '', timeout: 60, trackingSubmodules: true], [$class: 'RelativeTargetDirectory', relativeTargetDir: 'server-core'],[$class: 'CheckoutOption', timeout: 60]], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CalibrationResults.git']]])
                }
            }
            stage('Combination'){

                steps{
                    echo 'Checking out server spoke'
                    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false, timeout: 60], [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: false, recursiveSubmodules: true, reference: '', timeout: 60, trackingSubmodules: true], [$class: 'RelativeTargetDirectory', relativeTargetDir: 'server-spoke'], [$class: 'CheckoutOption', timeout: 60]], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CombinationBuilder.git']]])


                }
            }
        }

    }

}
}

Generated using Checkout git snippet generator

Answer 5

You can just add a shell step that does the cloning and moves it to a subdirectory of your choice.

git clone https://$bitbucketUsername:$bitbucketPassword@<yourbitbucketserver>/scm/projectname/reponame1.git
mv reponame1 new_subdir_name1

git clone https://$bitbucketUsername:$bitbucketPassword@<yourbitbucketserver>/scm/projectname/reponame2.git
mv reponame2 new_subdir_name2