安装 Big Sur 后，无法再通过 HTTPS 克隆大型存储库 git

Question

Whenever I try to clone large repos, this is what happens:

$ git clone https://github.com/yarnpkg/berry.git
Cloning into 'berry'...
remote: Enumerating objects: 60762, done.
remote: Counting objects: 100% (1155/1155), done.
remote: Compressing objects: 100% (588/588), done.
Receiving objects:   5% (3454/60762), 13.86 MiB | 4.60 MiB/ (etc etc)
fatal: fetch-pack: invalid index-pack output

I have no anti-virus installed, and I'm not on a VPN. I tried connecting to another network and that didn't solve it, so there must be something with Big Sur causing this. I'm not sure what else to try.

macOS 11.4, APFS+ filesystem, git 2.31.1

I've already tried changing compression settings, messing with pack size settings, this post did not help either. I'm posting this because I've tried everything else I've seen on the Internet so far and nothing has worked.

Answer 1

This should be a comment (as it's not an answer) but I need some formatting and a lot of space here. The short version is that you need to find out why git index-pack is misbehaving or failing. (Fetching over a smart protocol normally retrieves a so-called thin pack, which git fetch needs to "fatten" using git index-pack --fix-thin .)

The "invalid index-pack output" error occurs if the output from git index-pack does not match what git fetch-pack expects. Here's the code involved :

char *index_pack_lockfile(int ip_out, int *is_well_formed)
{
    char packname[GIT_MAX_HEXSZ + 6];
    const int len = the_hash_algo->hexsz + 6;

    /*
     * The first thing we expect from index-pack's output
     * is "pack\t%40s\n" or "keep\t%40s\n" (46 bytes) where
     * %40s is the newly created pack SHA1 name.  In the "keep"
     * case, we need it to remove the corresponding .keep file
     * later on.  If we don't get that then tough luck with it.
     */
    if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') {
        const char *name;

        if (is_well_formed)
            *is_well_formed = 1;
        packname[len-1] = 0;
        if (skip_prefix(packname, "keep\t", &name))
            return xstrfmt("%s/pack/pack-%s.keep",
                       get_object_directory(), name);
        return NULL;
    }
    if (is_well_formed)
        *is_well_formed = 0;
    return NULL;
}

This is run from fetch-pack.c 's get_pack function , which runs git index-pack with arguments based on a lot of variables. If you run your git clone with environment variable GIT_TRACE set to 1, you can observe Git running git index-pack . The call to index_pack_lockfile here only happens if do_keep is set, which is based on args->keep_pack initially, but can become set if the pack header's hdr_entries value equals or exceeds unpack_limit (see around line 859).

You can control the unpack_limit value using fetch.unpackLimit and/or transfer.unpackLimit . The default value is 100. You might be able to use these to work around some problem with index-pack, maybe—but index-pack should not be failing in whatever way it is failing. Note that if you want to force git fetch to use git unpack-objects instead, you must also disable object checking ( fsck_objects ).

It could be interesting to run git index-pack directly, too, on the data retrieved by git fetch . (Consider installing a shell script in place of the normal git index-pack , where the script prints its arguments, then uses kill -STOP on its own process group, so that you can inspect the temporary files.)

Answer 2

Solved!

TL;DR: ulimit was limiting max file size on my filesystem. Why this was never an issue in Catalina, who knows.

I had this in my shell startup script:

ulimit -n 65536 65536

That second parameter was the bad one. It was limiting max file size to 32MB. Strangely, I have this exact same setting on Catalina and it imposes no limitation whatsoever. Someone must've provided this setting and I just copied it without understanding the implications.

I ran ulimit -n unlimited unlimited and now all is well!

Interesting notes on ulimit changes from Catalina to Big Sur:

I ran this on my Catalina install:

$ ulimit -a                                                   
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              65536
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       11136
-n: file descriptors                65536
$ mkfile 500m whoa                                             
$ 

Look at that, No problem creating a 500MB file. clearly ignoring ulimit's 65536-block file size limit.

Yet on Big Sur, with the same ulimit settings:

$ mkfile 500m whoa                                                                                                                               
[1]    22267 file size limit exceeded  mkfile 500m whoa