Jenkins Scan Multibranch Pipeline fails Host key verification failed

Question

Jenkins is failing with an SSH issue when it tries to scan the multibranch pipeline and ssh is giving "Host key verification failed":

[Wed Sep 21 01:10:09 PDT 2022] Starting branch indexing...
 > git --version # timeout=10
 > git --version # 'git version 2.31.1'
using GIT_SSH to set credentials jenkins@jenkinsserver SSH private key
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
 > /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh7000755047282481393.key
Verifying host key using manually-configured host key entries
 > git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/vol/git/cmb.git # timeout=10
ERROR: [Wed Sep 21 01:10:09 PDT 2022] Could not update folder level actions from source blueocean
[Wed Sep 21 01:10:09 PDT 2022] Finished branch indexing. Indexing took 0.13 sec
FATAL: Failed to recompute children of CertAccord
hudson.plugins.git.GitException: Command "git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/\
vol/git/cmb.git" returned status code 128:
stdout:
stderr: No ECDSA host key is known for jenkinsserver.corp.contoso.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2697)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2009)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2000)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteSymbolicReferences(CliGitAPIImpl.java:3675)
        at jenkins.plugins.git.AbstractGitSCMSource.retrieveActions(AbstractGitSCMSource.java:1152)
        at jenkins.scm.api.SCMSource.fetchActions(SCMSource.java:848)
        at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:598)
        at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
        at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
        at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
        at hudson.model.ResourceController.execute(ResourceController.java:107)
        at hudson.model.Executor.run(Executor.java:449)
Finished: FAILURE

I'm aware that this problem indicates the ssh client is receiving an ssh server signature that it doesn't recognize (eg it's not in known_hosts).

If I "su - jenkins" on the jenkinsserver I can manually run git and ssh without a problem:

$ git ls-remote --symref -- ssh://jenkins@jenkinsserver.corp.contoso.com/vol/git/cmb.git
ref: refs/heads/master  HEAD
f79a54e2233749e0f0a9cf01        HEAD
... snip ...
$ ssh jenkinsserver.corp.contoso.com date
Wed Sep 21 01:45:20 PDT 2022
$ grep jenkinsserver ~/.ssh/known_hosts
jenkinsserver.corp.contoso.com ecdsa-sha2-nistp256 AAAAE2VjZHN....

This means that there is some other known_hosts file that Jenkins is somehow telling ssh to use. I just can't figure out where that file is!

Since this is a branch scan it's running on the jenkin's server, right?

The jenkinsserver is a RHEL 8.6 system. Jenkins version 2.361.1-1.1

Answer 1

Found a solution for this. In Manage Jenkins > Configure Global Security > Git Host Key Verification Configuration I changed it to "Accept first connection". It was set to "Manually provide keys".