GitLab 和 LFS 推送丢失的文件

Question

I have a very old repository I have been using and at some point some LFS files have gone missing. Quite a few of them. When I try and push to a new repository in Gitlab, I get the following error

 GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual 

Other QA for git-lfs ( here , here ) indicate that usually some variant of git lfs push --all but I don't have a working repo with these files anymore and they are gone for good. Gitlab ignores pushing with --no-verify and still gives the error.

Running git lfs fetch --all gives a lengthy list of missing OID

[43cb9e6d1d15bb8d31af911aa69a15a67174c5df73dabc85294ce08198cac468] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it
[454907d530534af9cc95903820c0a632a851b45de98ba18e1de117b8a649f8ac] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it
[ce1314f0c4cb05f349540fa144d33faeb2281ae552cf75dc866a8350d90fd2ac] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it
[d5e8925d273cb00341f00d0f40b39f97cced1e833ef687de2d4663836e7f4e45] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it
...

Another post here contains scripts to remove LFS entirely by checking out every commit and smudging LFS files, but this seems highly detrimental to the repository moving forward. Still another indicates a path to remove the LFS file entirely ( here ).

All these issues seem to be suboptimal:

1. Removing LFS will grossly bloat the repo and the smudge is likely to fail for the files already broken, anyway
2. Git pull/push -all is not an option because the files are gone for good
3. Removing the LFS is not right because a version of the files exist today only the object in some branch in the past is missing, so all instructions on how to remove LFS will break the repo.

Is there a way to have GitLab not ignore no-verify or to efficiently filter out specific OID from the history? I don't mind if the files are missing forever but I had hoped to preserve history.

I know that I can run git log --all -p -S 43cb9e6d1d15bb8d31af911aa69a15a67174c5df73dabc85294ce08198cac468 to get the commit and file (though it takes 5-10min to run PER OID, so this would take many hours), but I don't know what to do with that.

Answer 1

I just solved the same issue by the following steps:

• I have got the dirty file relative path by git lfs ls-files .
• I downloaded BFG Repo Cleaner at https://rtyley.github.io/bfg-repo-cleaner/
• I run bfg passing the parameters --delete-files <DIRTY_FILE_PATH>

The BFG log reported me the following warning:

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 9edf1837 (protected by 'HEAD') - contains 1 dirty file : 
    - requirements/LMD-SE-10-D11/setupse10d11.exe (132 B )

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

<REPO_PATH>/..bfg-report/2021-07-30/14-12-39/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it, 
and then run the BFG on a fresh copy of your repo.

In order to deal with this warning, which prevented me from removing the dirty file at all, I cloned again the repo, but without mirroring it .

Now, as suggested by the log, I deleted the dirty file by a commit, and run again the BFG command with the same arguments.

Finally, I issued the commands:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

And forced the history rewriting by git push --force .

Result: the dirty LFS reference has finally gone!