使用包含特殊字符（例如“\”、“:”、“;”）的其他文件列表过滤文件在 Linux

Question

I'm trying to filter a file where the file I want to filter is the mygeneralfile.txt and my file where the filters are has the following name myfilterfile.txt The contents of both files are as follows:

employee@EMPLOYEE-PC:~$ cat myfilterfile.txt
2020-06-24 00:00:04,396
2020-06-24 00:00:04,510
2020-06-24 00:00:04,511
employee@EMPLOYEE-PC:~$

employee@EMPLOYEE-PC:~$ cat mygeneralfile.txt
[2020-06-24 00:00:04,265][] [INFO] [com.mycompany.library] |getanotherTableImportant USERABCDEFG
[2020-06-24 00:00:04,311][] [INFO] [com.mycompany.library] |getanotherTableImportant null
[2020-06-24 00:00:04,314][] [INFO] [com.mycompany.library] |getanotherTableImportant USER_NUMBER_TWO_1234567
[2020-06-24 00:00:04,396][] [INFO] [com.mycompany.library] |getanotherTableImportant BILLABONG_MASTER_USER
[2020-06-24 00:00:04,510][] [INFO] [com.mycompany.library] |getanotherTableImportant NINET_USER_350
[2020-06-24 00:00:04,511][] [INFO] [com.mycompany.library] |getanotherTableImportant USERABCDEFG
[2020-06-24 00:00:04,527][] [INFO] [com.mycompany.library] |getanotherTableImportant USERABCDEFG

The result I want to obtain is the following:

[2020-06-24 00:00:04,396][] [INFO] [com.mycompany] |getanotherTableImportant BILLABONG_MASTER_USER
[2020-06-24 00:00:04,510][] [INFO] [com.mycompany] |getanotherTableImportant NINET_USER_350
[2020-06-24 00:00:04,511][] [INFO] [com.mycompany] |getanotherTableImportant USERABCDEFG

and if it just show me these lines it would be much better

BILLABONG_MASTER_USER
NINET_USER_350
USERABCDEFG

I've seen that the awk command is useful for these cases and try to replicate it using the following command: awk 'FNR==NR { a [$NF]; next } ($NF in a)' myfilterfile.txt mygeneralfile.txt awk 'FNR==NR { a [$NF]; next } ($NF in a)' myfilterfile.txt mygeneralfile.txt , however it does not generate any output for me and I think it's because inside my myfilterfile.txt file it contains special characters such as - : ,

PS: The mygeneralfile.txt file weighs about 2GB and that's why I'm trying the awk command as it's faster than the grep command. I have little knowledge with the awk command so I would like you to detail every function that is being used to solve this problem.

Thank you very much Community!!!

Answer 1

With GNU grep , you can try the following command:

# grep -F -f myfilterfile.txt mygeneralfile.txt

The patterns are searched through the whole line. I you are paranoid and want to limit the search to the date-time field, try this slower version:

# sed 's/^/^./' myfilterfile.txt | grep -f - mygeneralfile.txt

I doubt you will get a faster awk script.

Answer 2

$ awk -F'[][ ]' 'NR==FNR{a[$0]; next} ($2" "$3) in a{print $NF}' myfilterfile.txt mygeneralfile.txt
BILLABONG_MASTER_USER
NINET_USER_350
USERABCDEFG

wrt I think it's because inside my myfilterfile.txt file it contains special characters such as -: , - no, it's not. None of those characters are special even in a regexp (except - if it was in the middle of a bracket expression within a regexp) and your awk command is doing string comparison anyway so even if they were regexp metachars that wouldn't apply in this case.