Jenkins Github Authentication fails sometimes?

Question

I have a peculiar problem with Jenkins & Github. We have our repos on Github and we use Jenkins to clone and build. When I start a build, it fails almost instantly due to

ERROR: Error cloning remote repo 'origin'
stderr: remote: Invalid username or password.

However, if I run the job again, it builds without any issues. I searched for problems like this and people usually have authentication problems due to misconfiguration and for them, error happens every time. But for me, it only happens the first time and for subsequent builds it almost always works (%95) as expected. Not sure what's causing this issue but any help is greatly appreciated.

• Jenkins version: 2.332.2 ([Updated from] 2.263.4 also had the same problem)
• OS version: MacOS Monterey 12.4 (Big Sur also had the same problem)
• All of the Git related plugins are also up to date.

Here is the log of a failed job:

Started by user d***
12:21:27 Connecting to https://api.github.com using github-app-***
Obtained Jenkinsfile_client from ***
Loading library c***shared-lib***@**shared***
Examining ***shared-lib***
Attempting to resolve **shared*** as a branch
Resolved **shared*** as branch **shared*** at revision ***cda***
The recommended git tool is: NONE
using credential github-app-***
 > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/***_utils***_upm@libs/***167***/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/***shared-lib***.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/***shared-lib***.git
 > git --version # timeout=10
 > git --version # 'git version 2.30.2'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress -- https://github.com/***shared-lib***.git +refs/heads/**shared***:refs/remotes/origin/**shared*** # timeout=10
Checking out Revision ***cda*** (**shared***)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ***cda*** # timeout=10
Commit message: "updated ***"
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Mac Server ABC (mm) in /Users/xxx/***_utils***_upm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-app-***
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/***/utils***.git
 > git init /Users/xxx/***_utils***_upm # timeout=10
Fetching upstream changes from https://github.com/***/utils***.git
 > git --version # timeout=10
 > git --version # 'git version 2.30.1 (Apple Git-130)'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress -- https://github.com/***/utils***.git +refs/heads/upm:refs/remotes/origin/upm # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- https://github.com/***/utils***.git +refs/heads/upm:refs/remotes/origin/upm" returned status code 128:
stdout: 
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/***/utils***.git/'

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:847)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:376)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Mac Server ABC (mm)
    ...

The very next 'Rebuild' was perfectly fine and built without any problems.

Answer 1

I was having the same issue, see logs below:

Branch indexing
15:01:36 Connecting to https://api.github.com using GitHub App created to manage GitHub PR Status Checks in the iOS repos. See also: https://github.com/organizations/redacted/settings/installations/27296132
Obtained Jenkinsfile from 164c6778f38fccc46f80d064e335355e2ffe34da
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /Users/ec2-user/.jenkins/workspace/ches-multibranch-pipeline_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM) (hide)
[Pipeline] checkout
The recommended git tool is: git
using credential jenkins-github-app
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/redacted/redacted-ios.git
 > git init /Users/ec2-user/.jenkins/workspace/ches-multibranch-pipeline_master # timeout=10
Fetching upstream changes from https://github.com/redacted/redacted-ios.git
 > git --version # timeout=10
 > git --version # 'git version 2.32.0 (Apple Git-132)'
using GIT_ASKPASS to set credentials GitHub App created to manage GitHub PR Status Checks in the iOS repos. See also: https://github.com/organizations/redacted/settings/installations/27296132
 > git fetch --no-tags --force --progress -- https://github.com/redacted/redacted-ios.git +refs/heads/master:refs/remotes/origin/master # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- https://github.com/redacted/redacted-ios.git +refs/heads/master:refs/remotes/origin/master" returned status code 128:
stdout: 
stderr: remote: Invalid username or password.

I solved this by SSH'ing into Jenkins' machine and deleting credential.helper=osxkeychain from all git configs:

1. SSH into your Jenkins machine
2. Run git config --get-all --show-origin credential.helper to find where credentials.helper is at (all of its places). It will print something like this:

file:/Applications/Xcode-13.2.1.app/Contents/Developer/usr/share/git-core/gitconfig     osxkeychain

3. Edit the file and removed its contents:

nano /Applications/Xcode-13.2.1.app/Contents/Developer/usr/share/git-core/gitconfig

(don't forget to save)

4. Confirm that it was deleted by running git config --get-all --show-origin credential.helper again, which should now show empty results.

References:

• How to disable osxkeychain as credential helper in git config?
• https://serverfault.com/questions/823152/how-do-i-change-jenkins-git-plugin-from-using-git-askpass-to-gitcredentials/824596#824596
• https://github.com/jenkinsci/github-branch-source-plugin/pull/326

Hope this helps some people out there! I spent several days on this haha