Bash script to list files periodically

Question

I have a huge set of files, 64,000, and I want to create a Bash script that lists the name of files using

ls -1 > file.txt

for every 4,000 files and store the resulted file.txt in a separate folder. So, every 4000 files have their names listed in a text files that is stored in a folder. The result is

folder01 contains file.txt that lists files #0-#4000

folder02 contains file.txt that lists files #4001-#8000

folder03 contains file.txt that lists files #8001-#12000

.

.

.

folder16 contains file.txt that lists files #60000-#64000

Thank you very much in advance

Answer 1

You can try

ls -1 | awk '
{
    if (! ((NR-1)%4000)) {
        if (j) close(fnn)
        fn=sprintf("folder%02d",++j)
        system("mkdir "fn)
        fnn=fn"/file.txt"
    }
    print >> fnn
}'

Explanation:

• NR is the current record number in awk, that is: the current line number.
• NR starts at 1, on the first line, so we subtract 1 such that the if statement is true for the first line
• system calls an operating system function from within awk
• print in itself prints the current line to standard output, we can redirect (and append) the output to the file using >>
• All uninitialized variables in awk will have a zero value, so we do not need to say j=0 in the beginning of the program

Answer 2

This will get you pretty close;

ls -1 | split -l 4000 -d - folder

Run the result of ls through split , breaking every 4000 lines ( -l 4000 ), using numeric suffixes ( -d ), from standard input ( - ) and start the naming of the files with folder .

Results in folder00 , folder01 , ...

Answer 3

Here an exact solution using awk :

ls -1 | awk '  
 (NR-1) % 4000 == 0 {            
    dir = sprintf("folder%02d", ++nr)
    system("mkdir -p " dir);            
 }                                      
 { print >> dir "/file.txt"} '

Answer 4

There are already some good answers above, but I would also suggest you take a look at the watch command. This will re-run a command every n seconds, so you can, well, watch the output.