gerrit to gerrit replication issue

Question

I have got a gerrit 2.8.6 currently live, and I am about to roll out a complete new infrastructure based on gerrit 2.11. To verify the functionality of the new infrastructure, I wanted to setup gerrit replication from the live server to the new one, so that jobs would be triggered in the (new) jenkins and all the rest of the pipeline would be tested in parallel.

The replication plugin is setup for force pushing everything in the live server like this:

[remote "dev-infra"]
url = ansible@new.server.fqdn:${name}.git
push = +refs/*:refs/*

After rsyncing the filesystem with the repos, migrating the database, and executing init and reindex, I started the replication, but it started failing with this stacktrace on the live end:

[2016-01-07 11:57:01,561] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to ansible@new.server.fqdn:interface/ui.git
org.eclipse.jgit.errors.TransportException: ansible@ew.server.fqdn:interface/ui.git: Missing unknown 77752e00818523aa58fa61d64bdb8c059ab5d00d
    at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:217)
    at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152)
    at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165)
    at org.eclipse.jgit.transport.Transport.push(Transport.java:1173)
    at org.eclipse.jgit.transport.Transport.push(Transport.java:1219)
    at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:377)
    at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:351)
    at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:267)
    at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:78)
    at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:240)
    at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:237)
    at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
    at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
    at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75)
    at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:237)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
    at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 77752e00818523aa58fa61d64bdb8c059ab5d00d
    at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)
    at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
    at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)
    at org.eclipse.jgit.internal.storage.pack.PackWriterBitmapWalker.findObjects(PackWriterBitmapWalker.java:95)
    at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:1847)
    at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1665)
    at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:797)
    at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:760)
    at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:300)
    at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:197)
    ... 23 more

and this related stacktrace on the receiving end:

[2016-01-07 11:58:02,328] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user Ansible account 999999) during git-receive-pack 'interface/ui.git'
com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:176)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Unpack error on project "interface/ui":
  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@14c7d35cclass org.eclipse.jgit.transport.AdvertiseRefsHookChain

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:175)
        ... 12 more
Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:239)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:160)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)
        ... 12 more
Caused by: java.io.EOFException: Packfile is truncated.
        at org.eclipse.jgit.transport.PackParser.fill(PackParser.java:1138)
        at org.eclipse.jgit.transport.PackParser.readPackHeader(PackParser.java:838)
        at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:476)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:194)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePack(BaseReceivePack.java:1020)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:848)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:191)
        ... 14 more

What does missing unknown mean? If I try to execute git show with the ref on the project git repository, I get:

# git show 77752e00818523aa58fa61d64bdb8c059ab5d00d
fatal: bad object 77752e00818523aa58fa61d64bdb8c059ab5d00d

What can it be causing this behaviour? I found this very similar bug report, but the versions we use do not seem to be affected.

EDIT

# git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (459889/459889), done.
Checking connectivity: 459546, done.
dangling commit 3771a40494e90d369015e3de06d120c0ad146059
dangling commit 8368c5b1a95466b8b14fd24e45cea5e27b3cf6f9
dangling commit c111e6fca5a917b0ee3cb284cb3873289befd5f8
dangling commit 84a666f2181f4ab2587a4da0f7dc4ade24b4e45a

Answer 1

After executing the git fsck --full as suggested you mentioned that you have indeed dangling commits.

This mean that you repository is "broken" and you will need to clone it again from any other "good" location that you have a copy.

Right now your repository is in a broken state and basically is not usable.