我如何循环浏览特定数量的文件夹

Question

I am trying to loop through a folder that has 744 sub directories. How do i loop through only certain number of folders. Since i have 744 sub directories i would split this into half and loop through first 372 directories and then later on loop through rest of the 372 directories. I want to make sure i don't copy directories multiple times. Below is what i tried doing but i want to know what would be the effective way of doing this to avoid duplication.

for d in `ls -tr|tail -372`
do
echo $d
done

Answer 1

Since my xargs answer didn't receive any feedback, here's another approach.

printf "%s\n" */ |
awk 'BEGIN { n=1; OFS="\t"
        split("first:second:third", destination, /:/) }
    (i++ % 372)=0 { ++n }
    { print destination[n], $0 }'

This will add a field in front of each directory name, which you can use to process the results further. Sample output:

first   directory1/
first   directory2/
first   directory3/
:
first   directory372/
second  directory373/
second  directory374/
:
second  directory743/
second  directory744/

So the field value third from the Awk script is never used, but I put it in anyway to demonstrate that this could easily be extended to do three-way partitions, or four-way or what have you.

You would use this eg by piping to

while IFS= read -r dest dir; do
    echo mv "dir" "$dest"
done

Unlike the xargs -0 answer, this is not robust against arbitrary file names; in particular, directory names which contain newlines will not work correctly.

Actually a much better solution would be to split the files the other way -- ie for a two-way partition, print first on every other line, and second on every other. Then you don't have to hard-code the number of items, just the number of partitions.

printf "%s\n" */ |
awk 'BEGIN { OFS="\t"
        n = split("ernie:bert", host, /:/) }
    { print host[1+((NR-1)% n)], $0 }' |
while IFS= read -r server dir; do
    mkdir -p "$server"
    mv "$dir" "$server/"
done

Regardless of the number of directories, this splits them evenly into the directories ernie and bert , on the optimistic assumption that you (too) might have named your file servers after Sesame Street characters.

If you want to scp the directories instead of mv them, grouping them by server name would be a lot more efficient; but a simple sort takes care of that if necessary. (That's not the only reason we print the destination before each file name; it's also useful because then we don't have to worry that the directory names could contain our field separator.)

Answer 2

You can use xargs but this requires the 372 directory names to fit in one invocation (ie the directory names combined must not exceed ARG_MAX ).

printf '%s\0' */ |
xargs -n 372 -r -0 sh -c '
    d=dest$$; mkdir "$d"; cp "$@" "$d"' _

This will generate a unique new directory with the prefix dest and a number for each batch of directories it copies. There are probably better ways to split the files (and calling sh from xargs is not exactly a newbie-friendly answer) but maybe this should at least give you some ideas.

In some more detail, xargs -n 372 limits the number of arguments that get processed in one go, and the command you pass to xargs could be something a lot simpler; xargs -n 372 cp -t fnord would copy first 372 directories to fnord , then another 372; but in order for this to be actualy useful, we want the destination directory to change each time we call xargs , and so I put in a simple script which does that.

You also need to understand that 372 is a maximum, and if the directory names are really long, xargs could decide that it needs to pass fewer directories in order to not cross the "argument list too long" limit. But for your use case, on any remotely modern system, we are probably far below that limit anyway.

xargs -0 and cp -t are GNU extensions, ie they should work on Linux out of the box, and you can install them on most other platforms; if you really need to support something like Solaris without installing external tools, that's going to be slightly more challenging.

---

Addendum: Here's a xargs implementation of the ernie & bert part of the Awk answer:

printf "%s\0" */ |
xargs -r -0 -n 2 sh -c '
    mkdir -p ernie bert
    mv "$1" ernie
    mv "$2" bert' _

There will be an ugly but harmless error message for the last item if you have an uneven number of input directories. There are obvious but inelegant ways to fix that, or elegant but obscure ones; but I prefer to keep this plain for now.