du如何估算文件大小？

Question

I am downloading a large file with wget , which I ran in the background with wget -bqc . I wanted to see how much of the file was downloaded so I ran

du -sh *

in the directory. (I'd also be interested to know a better way to check wget progress in this case if anyone knows...) I saw that 25 GB had been downloaded, but for several attempts afterwards it showed the same result of 25 GB. I became worried that du had somehow interfered with the download until some time later when du showed a result of 33 GB and subsequently 40 GB.

In searching stackoverflow and online, I didn't find whether it is safe to use du on files being written to but I did see that it is only an estimate that can be somewhat off. However, 7-8 GB seems like a lot, particularly because it is a single file, and not a directory tree, which it seems is what causes errors in the estimate. I'd be interested to know how it makes this estimate for a single file that is being written and why I would see this result.

Answer 1

The operating system has to go guarantee safe access.

du does not estimate anything. the kernel knows the size of the file and when du asks for it that's what it learns.

If the file is in the range of gigabytes and the reported size is only with that granularity, it should not be a surprise that consecutive invocations show the same size - do you expect wget to fetch enough data to flip to another gigabyte in between your checks? You can try running du without sh in order to get a more accurate read.

Also wget will hold some amount of data in ram, but that should be negligible.

Answer 2

du doesn't estimate, it sums up. But it has access to some file-system-internal information which might make its output be a surprise. The various aspects should be looked up separately as they are a bit too much to explain here in detail.

1. Sparse files may make a file look bigger than it is on disk.
2. Hard links may make a directory tree look bigger than it is on disk.
3. Block sizes may make a file look smaller than it is on disk.

du will always print out the size a directory tree (or several) actually and really occupy on disk. Due to various facts (the three most common are given above) this can be different from the size of the information stored in theses trees.