Git fetch tags but missing latest version

Question

Recently I upgrade my gitlab( 8.17.3 -> 12.10.9 ). Everything works fine after upgrading git.

But occasionally, some project cannot fetch full tags(missing latest version). When working with jenkins, the error message is "Couldn't find any revision to build. Verify the repository and branch configuration for this job".

git clients version are: 1.7.1.2, 1.8.3.1, 2.8.4 and 2.22.0

How to reproduce:

git init test ;
cd test ;
git fetch --tags --progress http://url_to_git/abc.git "+refs/heads/*:refs/remotes/origin/*"  --prune

...
From From http://url_to_git/abc.gi
 * [new branch]      Dev_Long_Br -> origin/Dev_Long_Br
 * [new branch]      ft_dev_0805_ppt -> origin/ft_dev_0805_ppt
 * [new branch]      ft_dev_0911 -> origin/ft_dev_0911
 * [new branch]      master     -> origin/master
 * [new tag]         dev_fiba_dmm_20190302_1028 -> dev_fiba_dmm_20190302_1028
 * [new tag]         rls_fiba_dmm_20191126 -> rls_fiba_dmm_20191126

The result seems that missing latest branch "rls_0929 ". But wait for a while to retry fetch tags, I got the latest branch

git fetch --tags --progress http://url_to_git/abc.git "+refs/heads/*:refs/remotes/origin/*"  --
...
 From http://url_to_git/abc.git
   7501d7d..1a9a602  ft_dev_0911 -> origin/ft_dev_0911
 * [new branch]      rls_0929   -> origin/rls_0929

If I remove .git directory, redo fetch tags , everything seems to be right.

Work step is as follows:

rm -rf .git
retry fetch
git init; git fetch --tags --progress http://url_to_git/abc.git "+refs/heads/*:refs/remotes/origin/*"  --prune
...

 From From http://url_to_git/abc.gi
 * [new branch]      Dev_Long_Br -> origin/Dev_Long_Br
 * [new branch]      ft_dev_0805_ppt -> origin/ft_dev_0805_ppt
 * [new branch]      ft_dev_0911 -> origin/ft_dev_0911
 * [new branch]      master     -> origin/master
 * [new branch]      rls_0929   -> origin/rls_0929
 * [new tag]         dev_fiba_dmm_20190302_1028 -> dev_fiba_dmm_20190302_1028
 * [new tag]         rls_fiba_dmm_20191126 -> rls_fiba_dmm_20191126

I use nfs as the storage service, is it a storage cache problem nfs server info：

 # rpm -qa|grep nfs
libnfsidmap-0.25-15.el7.x86_64
nfs-utils-1.3.0-0.33.el7_3.x86_64
[root@gitpprdtools-prod34 ~]# service nfs status
Redirecting to /bin/systemctl status  nfs.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sat 2020-07-18 00:14:18 CST; 2 months 12 days ago
 Main PID: 16701 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

NFS client/gitlab app server info:

mount|grep git-data
NFS_IP:/var/opt/gitlab/git-data on /var/opt/gitlab/git-data type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=
sys,mountaddr=NFS_IP,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=NFS_IP)

The branch actually exists on remote git server. I doubt that is it a nfs problem, My git server is 4 git app server with one vip and mount same nfs storage. When error happens, the git server which client really request that has the nfs cache which not having the latest branch. So I modify my setting as git doc, but error continues https://docs.gitlab.com/12.10/ee/administration/high_availability/nfs.html .

In jenkins, when specify branch to build and the missing lastest branch error happens, the error is as followed:

Started from command line by jenkinsadmin
Building remotely on docker10-2.64 (C++ docker10 docker) in workspace /data/jenkins/workspace/workspace/Flow_dockerMakePackage_5387053_774
Cloning the remote Git repository
Cloning repository http://git.a.com/b/c.git
 > git init /data/jenkins/workspace/workspace/Flow_dockerMakePackage_5387053_774 # timeout=10
Fetching upstream changes from http://git.a.com/b/c.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials root_gitlab
 > git fetch --tags --progress http://git.a.com/b/c.git +refs/heads/*:refs/remotes/origin/* # timeout=30
 > git config remote.origin.url http://git.a.com/b/c.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url http://git.a.com/b/c.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
 > git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://git.a.com/b/c.git
using GIT_ASKPASS to set credentials root_gitlab
 > git fetch --tags --progress http://git.a.com/b/c.git +refs/heads/*:refs/remotes/origin/* --prune # timeout=30
 > git rev-parse b215fc6cf5a4f431408773e3e05130a72f263546^{commit} # timeout=10
 > git rev-parse origin/b215fc6cf5a4f431408773e3e05130a72f263546^{commit} # timeout=10
 > git rev-parse b215fc6cf5a4f431408773e3e05130a72f263546^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

Can anyone help?

Answer 1

You describe the normal behavior of git with a rls_0929 branch that doesn't exist at first on the remote, and is created between your 2 git fetch invocations.

What elements lead you to think otherwise?

To see if the branch you expect actually exists on your remote, inspect its content:

• through gitlab's GUI (you tagged gitlab because your remote is a gitlab instance, correct?)
• from the cli: git ls-remote <url> refs/heads/rls_0929