git gc: no space left on device, even though 3GB available and tmp_pack only 16MB

Question

> git gc --aggressive --prune=now
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66685/66685), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_cO6T53' write error: No space left on device

sigh, ok

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19G   15G  3.0G  84% /
udev            485M  4.0K  485M   1% /dev
tmpfs            99M  296K   99M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            494M     0  494M   0% /run/shm
cgroup          494M     0  494M   0% /sys/fs/cgroup

doesn't look that bad

ls -lh .git/objects/pack/
total 580M
-r--r--r-- 1 foouser root  12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r-- 1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r-- 1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r-- 1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r-- 1 foouser root  11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r-- 1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r-- 1 foouser root 1.6M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r-- 1 foouser root 290M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r-- 1 foouser root  40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r-- 1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r-- 1 foouser root 1.6M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r-- 1 foouser root 102M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r-- 1 foouser root 1.6M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r-- 1 foouser root 151M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r-- 1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r-- 1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r-- 1 foouser root 6.2K Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r-- 1 foouser root 4.2M Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r-- 1 root          root  16M Feb 27 08:19 tmp_pack_cO6T53

So, git gc bails out on a tmp pack that's only 16MB big while my disk appears to have 3GB free. What am I missing? How can I get git gc to work more reliably? I've tried without aggressive option and --prune instead of --prune=now as well, same story.

Update

Doing a df -h during the repack action it shows that it is now using all my disk (100% usage). A little while later the repack action fails and it leaves another 14MB file in the .git/objects/pack/ folder. So, to recap, my packs use a total of 580MB. git repack somehow manages to use up 3GB to repack that. I have ~800MB free in the RAM after it's done btw. - maybe it's using so much working memory that it clogs up the swap? I guess my question comes down to: Are there options to make git repack less resource hungry?

versions: git version 1.7.9.5 on Ubuntu 12.04

Update 2 I've updated git to 2.3. Didn't change anything unfortunately.

> git --version
git version 2.3.0
> git repack -Ad && git prune
Counting objects: 68752, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (36893/36893), done.
fatal: sha1 file '.git/objects/pack/tmp_pack_N9jyVJ' write error: No space left on device

Update 3

Ok, so I just noticed something that is curious: the .git directory actually uses much more disk space than the 508MB previously reported.

> du -h -d 1 ./.git
8.0K    ./.git/info
40K ./.git/hooks
24M ./.git/modules
28K ./.git/refs
4.0K    ./.git/branches
140K    ./.git/logs
5.0G    ./.git/objects
5.0G    ./.git

Upon further inspection .git/objects/pack actually uses 4.5GB. The differences lies in hidden temp files I didn't notice before:

ls -lha ./.git/objects/pack/
total 4.5G
drwxr-xr-x   2 foouser root  56K Feb 27 15:40 .
drwxr-xr-x 260 foouser root 4.0K Oct 26 14:24 ..
-r--r--r--   1 foouser root  12K Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.idx
-r--r--r--   1 foouser root 5.1M Oct 30 05:47 pack-0301f67f3b080de7eb0139b982fa732338c49064.pack
-r--r--r--   1 foouser root 5.1K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.idx
-r--r--r--   1 foouser root 100K Oct 14 10:51 pack-27da727e362bcf2493ac01326a8c93f96517a488.pack
-r--r--r--   1 foouser root  11K Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.idx
-r--r--r--   1 foouser root 2.6M Oct 25 10:35 pack-4dce80846752e6d813fc9eb0a0385cf6ce106d9b.pack
-r--r--r--   1 foouser root 1.6M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.idx
-r--r--r--   1 foouser root 290M Apr  3  2014 pack-4dcef34b411c8159e3f5a975d6fcac009a411850.pack
-r--r--r--   1 foouser root  40K Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.idx
-r--r--r--   1 foouser root 6.1M Oct 26 11:53 pack-87529eb2c9e58e0f3ca0be00e644ec5ba5250973.pack
-r--r--r--   1 foouser root 1.6M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.idx
-r--r--r--   1 foouser root 102M Apr 19  2014 pack-9d5ab71d6787ba2671c807790890d96f03926b84.pack
-r--r--r--   1 foouser root 1.6M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.idx
-r--r--r--   1 foouser root 151M Oct  3 10:12 pack-af6562bdbbf444103930830a13c11908dbb599a8.pack
-r--r--r--   1 foouser root 4.7K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.idx
-r--r--r--   1 foouser root 125K Oct 20 11:02 pack-c0830d7a0343dd484286b65d380b6ae5053ec685.pack
-r--r--r--   1 foouser root 6.2K Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.idx
-r--r--r--   1 foouser root 4.2M Oct  2 15:38 pack-c20278ebc16273d24880354af3e395929728481a.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:37 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.idx
-r--r--r--   1 root          root  14M Feb 27 15:29 .tmp-7729-pack-00447364da9dfe647c89bb7797c48c79589a4e44.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:32 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.idx
-r--r--r--   1 root          root  41M Feb 27 15:30 .tmp-7729-pack-020efaa9c7caf8b792081f89b27361093f00c2db.pack
-r--r--r--   1 root          root 1.1K Feb 27 15:37 .tmp-7729-pack-051980133b8f0052b66dce418b4d3899de0d1342.idx
(continuing for a *long* while). 

Now I'd like to know: Is it safe to just delete those?

Answer 1

So here is what I found out so far: I couldn't find any documentation about these hidden '.tmp-XXXX-pack' in the .git/objects/pack folder. All other threads I can find are about non-hidden files with tmp_ prefix in the same folder. The hidden ones are also clearly created during the repack action and it's possible that these get stuck as well. I can't confirm whether that's still possible in git 2.3.0 (which I've updated to since), but at least the disk space requirement doesn't seem to have changed in this newer version - it still can't complete gc/repack. By deleting these .tmp-files I was able to recover my last 4GB and git still seems to behave fine afterwards - your results may vary though, so please make sure you have a backup before doing this . Finally, even 4GB wasn't enough to repack with gc --agressive . My .git folder is 1.1GB after the cleanup, my entire repository is 1.7GB. So 2x the size of your repository is possibly not enough for git gc , even with the aggressive option (which should save space). So I had to recover more space from elsewhere first.

Finally, here is what I have in my cleanup script now (which I think might be a good idea to call from a cron job):

#!/bin/bash
set -e

#git gc or remove tmp if that fails (because out of disk space)
git gc --aggressive --prune=now || rm -f .git/objects/*/tmp_* && rm -f .git/objects/*/.tmp-*

Answer 2

Similar scenario (about 2.3G available), except git gc itself would also fail with fatal: Unable to create '/home/ubuntu/my-app-here/.git/gc.pid.lock': No space left on device

What worked was to git prune first, and then run the gc.