Git hangs while pulling from repository on shared folder using Jenkins

Question

I am using Jenkins slave in my jobs, the slaves are amazon spot instances so I am using shared folder (EFS) in order to mount shared folders such .m2/.npm and the workspace.

when the job starts and tries to pull from remote git repository the build hangs while cloning.

When I am not using the efs and clone on the spot instance itself everything works as expected, creating other files manually or through Jenkins on the efs also works fine. permissions for the shared folder is the same as the user Jenkins uses.

Any suggestion what can cause this behavior?

This is the build log:

11:41:20 Fetching upstream changes from git@git.assembla.com:alpha.saas.git
11:41:20  > git --version # timeout=10
11:41:20 using GIT_SSH to set credentials jenkins@Dev_Builder(ssh)
11:41:20  > git fetch --no-tags --progress git@git.assembla.com:alpha.saas.git +refs/heads/*:refs/remotes/origin/* # timeout=5
11:41:29  > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
11:41:29  > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
11:41:29 Checking out Revision e0dd60499d693a40fa0d3669201437b49cc2b0c4 (refs/remotes/origin/master)
11:41:29  > git config core.sparsecheckout # timeout=10
11:41:29  > git checkout -f e0dd60499d693a40fa0d3669201437b49cc2b0c4
11:48:59 Build was aborted

Answer 1

So it seems that the problem is with large Packfile size in git, looks like the EFS is pretty slow handling such large files what made it hang for so long. I've noticed it by using the lsof command:

lsof +D ./

Which point it out the index file and the packfile are open for a long time and they both large files:

./.git/objects/pack/pack-601f9b58380bc69d49bcc429d046c8940c5ed9d2.idx

./.git/objects/pack/pack-601f9b58380bc69d49bcc429d046c8940c5ed9d2.pack

What I did in order to resolve it was using a shallow clone in Jenkins:

checkout([
        $class: 'GitSCM',
        branches: [[name: "$git_branch" ]],
        doGenerateSubmoduleConfigurations: false,
        extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: true]],
        submoduleCfg: [],
        userRemoteConfigs: [[url: "$git_repo" , credentialsId: env.gitCredentialsJenkins]]])

That solved the issue though it still takes to long to clone a 1.8GB repository.