bash / bin / grep：参数列表太长（使用--file选项）

Question

I have a text file containing 33.869 rows and I have to filter 30.067 of them.

With an example:

File: input.txt (csv like with 33.869 rows)

#00001:A123456.10.101.102,first,row,value2,1
#00002:A123456.10.101.103,second,row,value7,85
(omissis)
#33869:A123456.25.170.180,last,test,value9,0

File: filter.txt (list of values separated by "\\n" with 30.067 rows)

A123456.10.101.102
A123456.10.101.103
(omissis)
A123456.24.150.115

(expected) Output file: output.txt (csv like with 30.067 rows taken from input.txt):

#00001:A123456.10.101.102,first,row,value2,1
#00002:A123456.10.101.103,second,row,value7,85
(omissis)
#30067:A123456.24.150.115,whatever,x,y,99

The command I'm using is:

#!/bin/bash
/bin/grep --file="filter.txt" input.txt > output.txt

but error returned is

/bin/grep: Argument list too long

Am I forced to split "filter.txt" in smaller chunk?

Which is the limit allowed?

I did not find the limit on man code command.

Answer 1

If there are no regular expressions in the input file, you should switch to grep -F which can read a significantly larger number of input records.

Failing that, splitting the input file would be hugely more efficient than running 30,000+ iterations of grep over the same file.

Here's splitting in chunks of 10,000 lines; adapting to a different factor should be trivial.

#!/bin/sh

t=$(mktemp -d -t fgrepsplit.XXXXXXXXXXXX) || exit
trap 'rm -rf "$t"' EXIT       # Remove temp dir when done
trap 'exit 127' HUP INT TERM  # Remove temp dir if interrupted, too

split -l 10000 "$1" "$t"/pat

for p in "$t"/pat*; do
    grep -F -f "$p" "$2"
done

Answer 2

From what you write, I wonder whether grep is the right tool for the job. With grep you would usually try to apply a small set of matching rules, expressed as regular expressions. In your case, you match against a long list of literals.

This seems to be a case of finding the lines that full_file.txt and filtered.txt have in common. You might want to look at the following tools to achieve this:

• join ( http://linux.die.net/man/1/join ) gives you the lines that two files have in common. Note that both files have to be sorted. You can use process substitution to achieve this.
• combine ( http://linux.die.net/man/1/combine ) is a more general utility that does not require the input to be sorted. But it may not be available everywhere.

Answer 3

What about iterate on each line of your file ? something like :

while IFS= read -r  i ; do
   grep "$i" full_file.txt
done < grep_filter.txt >filtered.txt

Answer 4

用awk ：

awk -F"[:,]" 'FNR==NR{a[$2]=$0;next} ($0 in a) {print a[$0]}'  input.txt filter.txt