Why readdir() call in linux grows non-linearly

Question

I have a directory with 1000 files and readdir() takes less than 1 sec, but 10000 files take around 24 sec.

Why? It should be linear.

Can anyone explain the reason. And is there a better solution if only I need is to get the file and sub-directory names in a directory?

EDIT I am on my local linux pc.

Answer 1

It might be file system specific. Perhaps using a suitably configured Ext4 or BTRFS file system should help. Some file systems are using hashing or B-tree techniques to make the complexity of file access in a directory of size N be O(log N) , others are still linear eg O(N) , and the kernel might do weird things above that.

The shell that you might use in your huge directories will generally sort entries when globbing (see also glob(7) ). And you don't want its auto-completion to last many seconds on each keystroke!

I believe that you should never have huge directories (eg with more than a few hundred entries), so 10000 files in a single directory is unreasonable. If that is the case, you'll better organize your files differently, eg subdir01/file001.txt ... sbudir99/file999.txt

BTW, if your need is to have a lot of small things accessible by some textual key, using an indexed file (like gdbm ) or a Sqlite "database", or a real database ( PostGreSQL , MongoDb ...) is much more suitable, and probably more efficient. Don't forget to dump the data (probably in some textual format) for backup.

Notice that the documentation of readdir(3) on Linux, and of POSIX readdir do not mention any time complexity or any linear behavior. This lack of mention is significant. On the commonly used FAT filesystem (eg on many USB keys) the time complexity is probably quadratic.

Answer 2

It has no reason to be linear. At lower level, a directory is like a file, a collection of clusters. If it is contained in one single cluster, you have only one actual physical read operation, the remaining occurs in memory. But when you directory becomes excessively large, you will have many physical reads. At this moment, as stated by Basile Starynkevitch, it becomes highly dependent on the file system structure.

But IMHO, if you want to browse the directory, it depends essentially on the number of clusters used by the directory. It is much more implementation dependant when you directly look for a file (by name) in a huge directory. Filsystems with linear search will have worse results than filesystems using natively hashing like for example BSD FFS.

Answer 3

• All operations should be linear on a poor filesystem (eg FAT/FAT32 are O(N) ).
• Seeks, updates and deletes should be better than linear on a good filesystem like NTFS which is O(log N) . A full directory listing will still be linear though.
• In either case it should be much, much faster than what you have reported in both the small and large cases.

I suspect something else is going on. Very likely your results are biased by other factors than the directory structure, such as:

• Disk has a hardware problem which is triggered in the large example but not the small one
• Other disk activity from other parts of the system interrupts the test in the large case
• Disk hardware pre-fetching. Disks contain RAM caches which will try to predict which sectors will be requested next, and have them ready.
• Operating system cache. Operating systems will also cache data in a similar way.
• You are possibliy doing something with the data other than just readdir and this other operation has higher time complexity which dominates.
• Your application memory usage pattern is able to fit into L1 cache for small directories but not large ones.
• Your application memory usage pattern forces swapping on large directories but not small ones.

Answer 4

readdir is at best linear. If we ignore everything that goes on in the filesystem, the amount of data (file names and other stuff in struct dirent) from the kernel into userland is directly proportional to the number of files. So we start with O(n).

Then the kernel needs to figure out which data to give you. At best it is linearly stored in something that looks like a file. This is what older file systems like FFS and EXT2 do. This gives good performance for readdir (because finding which disk block to give you is just an array lookup), but has the disadvantage that actually opening those files ( open , stat or almost anything else that works with the file name) becomes an O(n) operation because every open has to linearly scan the directory to find the file name. This is why there has been so much work in caching directory data for those file systems. Even on those filesystems you might end up seeing that larger directories take longer to read per item because the way file information is stored gets more expensive with file size. Depending on your file (or directory) size the kernel might need to read between 1 and 5 other blocks from disk (or cache) to find out which block to give you.

If you have a different filesystem (most modern ones), they trade the convenience and speed of a linear directory for a more complex structure on disk which gives you a much better performance of open and stat (after all, why would you readdir if you don't intend to do anything with the files?), but as a consequence you end up (not necessarily, but most likely) with worse than linear time to actually perform a readdir because the operation to find out which disk block to read for your information might be O(log n).