Jenkins Pipeline: scm checkout shallow copy fails

Question

I am using Jenkins file to build a pipeline. I am trying to clone the reference repository using DSL like below.

checkout(
[$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, 
extensions: [[$class: 'CloneOption', depth: 1, noTags: false, reference: '', shallow: true]], 
submoduleCfg: [], 
userRemoteConfigs: [[url: 'git@bitbucket.org:user_team/infrastructure-as-code.git']])

and while the pipeline is being executed, it is being translated to this

git fetch --tags --progress git@bitbucket.org:userteam/infrastructure-as-code.git +refs/heads/*:refs/remotes/origin/* --depth=1

This clones the whole repository on my Jenkins server. I just want to obtain a shallow copy of my repo so that I could save my Jenkins server from space crunch. Please help here.

I am using: Jenkins version: 2.58,

Plugins:

Pipeline SCM Step: 2.4

Git: 3.3.0

Answer 1

I think you are misunderstanding the meaning of shallow clone.
Shallow clone will still clone the entire repository.
The difference will be that history will be truncated to the specified number of commits (in your case 1, since you have mentioned depth to be one.) It can save you a lot of space and time.

For more information please follow this link: git-clone#Documentation

For instance, see the below image where I am cloning same repository ( https://github.com/spring-cloud/spring-cloud-config.git ) 2 times, one without depth and one with depth=1. In first case, the local repository size is 40 MB and with depth the local repository size is mere 3.4 MB.

Answer 2

I would recommend to check https://issues.jenkins-ci.org/browse/JENKINS-43878 for better understanding. The reporter of this ticket compares the duration of clone+checkout process in 3 cases: non-shallow clone with git command, shallow clone with pipeline and shallow clone(depth=1) with git command, and the ticket reporter complains that case #2 lasts much longer than case #3.

I exercised with the repo https://github.com/tesseract-ocr/tessdata (~5 GB) and I could not reproduce the duration difference. But I observed the big size difference. These are my measurements:

1. Non-shallow clone with pipeline: 8 min, total size 4615 MB, "fetch size" 3256 MB.
2. Non-shallow clone with git command: 8 min, total size 4615 MB.
3. Shallow clone with pipeline: 4-5 min, total size 3121 MB, "fetch size" 1762 MB
4. Shallow clone(depth=1) with git command: 4-5 min, total size 1995 MB.

(the "fetch" size in my comparison is the size of the directory which I measured with "du -ms" at the moment after "git fetch" and before "git checkout" when it was done with the help of Jenkins pipeline)

If you compare cases 2 and 3 you will see that for shallow clone the approach "fetch+checkout" leads to more disk space occupation than for the normal "clone".

The pipeline maintainers agreed with this fact, but closed the ticket with "Won't fix", because they don't want to change the way of working from "fetch+checkout" to "clone" for the plugin due to other reasons.

This is exactly answer to your question why don't you see big difference between shallow and full clone for Jenkins pipeline: because Jenkins pipeline uses "fetch+checkout" approach which in case of --depth works differently than "clone" and downloads more data than "clone".

If you need a normal "clone --depth" it should be run as a shell command from the pipeline script. On my opinion it is a disadvantage of Jenkins pipeline.