How to feed a list of files into a bash script

Question

I'm having problems feeding a large list of filenames into a command in (git for windows git, I think it's cygwin)bash shell.

This is the basic command that works fine with a small set of arguments: git filter-branch -f --tree-filter 'rm -rf file1 directory2 file3 file4' However, I have about 1500 filenames.

I've tried: git filter-branch -f --tree-filter 'rm -rf file1 directory2... all 1500 names here' but I get an error:

/mingw64/bin/git: Argument list too long

I've tried to use a for loop: git filter-branch -f --tree-filter 'for f in $(cat files.txt); do rm -fr "$f"; done' git filter-branch -f --tree-filter 'for f in $(cat files.txt); do rm -fr "$f"; done' git filter-branch -f --tree-filter 'for f in $(cat files.txt); do rm -fr "$f"; done' and this runs through the loop with an error:

cat: files.txt: No such file or directory

FYI - the files.txt contents look like this:

./file1 
./directory2 
./file3 
./file4

Then I tried: git filter-branch -f --tree-filter < cat files.txt and cat files.txt | git filter-branch -f --tree-filter cat files.txt | git filter-branch -f --tree-filter but I get errors about the syntax not being correct - it shows the 'help' dialogue. eg:

usage: git filter-branch [--setup ] [--subdirectory-filter ] [--env-filter ] [--tree-filter ] [--index-filter ] [--parent-filter ] [--msg-filter ] [--commit-filter ] [--tag-name-filter ] [--original ] [-d ] [-f | --force] [--state-branch ] [--] [...]

Then I thought maybe I could just add the arguments into the file like this: git filter-branch -f

File:

--tree-filter './file1 ./directory2 ./file3 ./file4'

But I get the 'help' dialogue again.

I'm sure there is a way to do this, but my unix-fu is too weak. Please help!

---

In response to @dash-o:

I tried this, but am getting an error:

C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: eval: line 414: unexpected EOF while looking for matching `'' C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: eval: line 415: syntax error: unexpected end of file

The files.txt lists the files one per line. However, the rm -rf requires them to be on a single line and space-delimited.

I tried to put the files names on a single line, but I get a different error:

C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: line 414: rm -rf.vs./file1./directory2: command not found tree filter failed: 'rm -rf./file1./directory2'

Maybe the single quotes are being escaped and are not wrapped around the rm command?

Answer 1

Give this a try, just make sure you're in the path where files.txt is located to avoid the "No such file or directory" error:

for f in $(cat files.txt); do git filter-branch -f --tree-filter 'rm -rf '"$f"'' ; done

Answer 2

For cases of larger number of files, it might be time consuming to iterate over each file individually. For those cases, consider using xargs ability to batch arguments together (based on max number of argument, max command line size, etc).

xargs -L50 < files.txt  | xargs -I@ git filter-branch -f --tree-filter "'rm -rf @'"

The first xargs is used purely to arrange block arguments, 50 files each. This can be customized to include a line size limit, if needed.

The second xargs will execute git filter for each 50 arguments. You might need to work on the quoting, as I do not have the ability to test the command with git on windows).