Jenkins Git Plugin unable to initialize submodules

Question

I'm using Jenkins to build a project that has multiple Git submodules. While the main repository can be fetched, the submodule initialization is failing, even on a fresh clone:

Started by user anonymous
[EnvInject] - Loading node environment variables.
Building in workspace C:\Builds\Test Build
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git
 > C:\Program Files\Git\mingw64\bin\git.exe init C:\Builds\Test Build # timeout=10
Fetching upstream changes from git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git
 > C:\Program Files\Git\mingw64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Matt's Bitbucket SSH key
 > C:\Program Files\Git\mingw64\bin\git.exe fetch --tags --progress git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git +refs/heads/*:refs/remotes/origin/* # timeout=20
 > C:\Program Files\Git\mingw64\bin\git.exe config remote.origin.url git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git # timeout=10
 > C:\Program Files\Git\mingw64\bin\git.exe config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > C:\Program Files\Git\mingw64\bin\git.exe config remote.origin.url git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git # timeout=10
Fetching upstream changes from git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git
using GIT_SSH to set credentials Matt's Bitbucket SSH key
 > C:\Program Files\Git\mingw64\bin\git.exe fetch --tags --progress git@bitbucket.org:ACCOUNT/MAIN_REPOSITORY.git +refs/heads/*:refs/remotes/origin/* # timeout=20
 > C:\Program Files\Git\mingw64\bin\git.exe rev-parse "origin/master^{commit}" # timeout=10
Checking out Revision COMMIT_HASH (origin/master)
Commit message: "Updated Submodules"
 > C:\Program Files\Git\mingw64\bin\git.exe config core.sparsecheckout # timeout=10
 > C:\Program Files\Git\mingw64\bin\git.exe checkout -f COMMIT_HASH
First time build. Skipping changelog.
 > C:\Program Files\Git\mingw64\bin\git.exe remote # timeout=10
 > C:\Program Files\Git\mingw64\bin\git.exe submodule init # timeout=10
FATAL: Command "C:\Program Files\Git\mingw64\bin\git.exe submodule init" returned status code 128:
stdout: 
stderr: fatal: 'submodule' appears to be a git command, but we were not
able to execute it. Maybe git-submodule is broken?

hudson.plugins.git.GitException: Command "C:\Program Files\Git\mingw64\bin\git.exe submodule init" returned status code 128:
stdout: 
stderr: fatal: 'submodule' appears to be a git command, but we were not
able to execute it. Maybe git-submodule is broken?

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.submoduleInit(CliGitAPIImpl.java:945)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.setupSubmoduleUrls(LegacyCompatibleGitAPIImpl.java:81)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setupSubmoduleUrls(CliGitAPIImpl.java:71)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setupSubmoduleUrls(CliGitAPIImpl.java:1417)
    at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:100)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1188)
    at hudson.scm.SCM.checkout(SCM.java:495)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:560)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:485)
    at hudson.model.Run.execute(Run.java:1735)
    at hudson.matrix.MatrixBuild.run(MatrixBuild.java:313)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:405)
Finished: FAILURE

Running "git submodule init" from the Git Bash works as expected.
What might I be missing to make Jenkins behave correctly?

Configuration:

• Windows 7
• Jenkins 2.73.1
• Git Plugin 2.5.0
• Git Client Plugin 3.5.1
• Git for Windows 2.14.1.windows.1
• Main repository and submodules are private Git repositories in Bitbucket under the same account, and ssh keys have been exchanged.
• Advanced sub-modules behaviors options:
  • "Recurvisely udpate submodules"
  • "Use credentials from default remote of parent repository"

This appears different than either:

• 32947108 Unable to retrieve git submodule in jenkins with git-client plugin

• 35312274 Jenkins git submodule update fails

The latest tool versions appear to have the known submodule bugs fixed. I have tried the following without any effect:

1. Using Jenkins-stored SSH keys versus ~/.ssh local keys
2. Using HTTPS-based repo access instead of ssh
3. Using relative paths instead of absolute in .gitmodules
4. Using repos hosted on GitHub instead of BitBucket

Answer 1

I have built this project in Windows 10 and Linux (Ubuntu 17.04) to test. In both cases I have specified the SSH key directly in Jenkins and specifically removed it from ~/.ssh

Windows 10 fails to build, similar to above listing. Linux clones both the parent and submodule repos without issue.

Given these results, I conclude that the problem is specific to Windows + GitPlugin.

Answer 2

The git plugin is using the wrong git.exe. It has been configured to use git.exe from:

C:\Program Files\Git\Mingw64\bin\git.exe

The git.exe in that location apparently can't find the submodule command.

Configure the git plugin to use the git.exe from the default location and it will work as expected. The default location is:

C:\Program Files\Git\bin\git.exe

If I configure a Windows command prompt with the failing PATH settings, it reports the same message,

fatal: 'submodule' appears to be a git command but we were not able to execute it.

If I configure a Windows command prompt with the working PATH setting, the git submodule command behaves as expected.

Why This is Happening

It could be that the location of your git command has been defined (incorrectly) in Jenkins as

C:\Program Files\Git\Mingw64\bin\git.exe

It could be that the PATH environment variable includes that mingw64 directory.

Answer 3

did you verify that your git is the correct one ?

did you try - https://wiki.jenkins.io/display/JENKINS/SSH+Agent+Plugin ?