获取递归包含在目录中的文件列表的最快方法是什么？

Question

I have a directory that contains millions of files spread out in a hierarchy of folders. This directory is stored on a large remote NFS filesystem. I'd like to retrieve the list of these files as fast as possible.

Is it possible to go faster than find . > list.txt find . > list.txt ? What factors affect speed? I'm using python, but any solution will go as long as it's fast .

Answer 1

On linux, this was the fastest for me. Use (bash) globbing and printf like this:

printf "%s\n" directory/**/file
printf "%s\x00" directory/**/filename-with-special-characters | xargs -0 command

Seems to be a lot faster than

find directory -name file

or

ls -1R directory | grep file

or even, surprisingly,

ls directory/**/file

This was a local file system though: x86_64 system, ext4 filesystem on an SSD, in a directory structure of over 600,000 directories with multiple files in them.

Answer 2

Depending what do you want in the output. I recommend using

ls -R | grep ":$" | sed -e 's/:$//' -e 's/^/   /' -e 's/-/|/'

for the complete path to all the files recursively in the current directory.