如何测试 File2 中是否存在来自 File1 每一行的 substring

Question

I have two files with following data

file1:

6100540301SD01        ON5330399520191104906781            2019110390678151053303995ACK          20191105
6100540301SD01        ON0403096420191104225695            2019110322569551004030964A            20191105
6005260301SD01        46460045792019110490678911059455    2019110490678951000755694BE3        1120191105
6005260301SD01        46460045792019110490679616020577    2019110490679651000764053BDJDEDH    1620191105

file2:

20191104
20191105
20191106

Since file1 is fixed width file, the string at character position 97 to 104 is date. I want to extract the string by position from 97 to 104 and check if this exists in file2 . If it exists, I want to copy whole line to file3 .If its not, I want to copy it to file4 .

I have created C++ program but it is taking long time to process the file1 while is almost half million records. Therefore, if there is any awk/sed script that can be helpful, please share.

Answer 1

Turn the contents of file2 into a regular expression like 20191104|20191106|20191106 . Then you can use grep to match it.

patterns=$(<file2)
# Replace newlines with |
pattern=${patterns//$'\n'/|}
# Put ^.{96} at the beginning so it matches starting at column 97
pattern="^.{96}($pattern)"
grep -E "$pattern" file1 > file3 # Lines that match
grep -v -E "$pattern" file1 > file4 # Lines that don't match

If running grep twice is too slow, you could use awk :

awk -v pat="$pattern" '$0 ~ pat { print >>"file3"; next} {print >>"file4"}'

Answer 2

awk to the rescue!

$ awk 'NR==FNR {dates[$0]; next} 
               {print > (substr($0,97,104) in dates?"file3":"file4")}' file2 file1

Answer 3

This might work for you (GNU sed):

sed 's#.*#/^.\\{96\\}&/ba#' file2 | sed -nf - -e 'w file4' -e 'b;:a;w file3' file1

Create a script from file2 which writes each match to file3 and any remaining lines to file4.

The first invocation of sed passes its output to the second invocation of sed which in turn is supplemented with a couple of strings of commands inline. All matches are sent to the loop holder :a which writes them out to file3 any that are not matched, fall through to be written to file4.