删除文件中与其他文件中任何行匹配的所有行的 Grep 方法？

Question

I have a file of camera information where each line has a unique ID of the format

{"_id":{"$oid":"5b0cfa5845bb0c0004277e13"},"geometry":{"coordinates":[139.751,35.685]},"addEditBy":["dd53cbd9c5306b1baa103335c4b3e91d8b73386ba29124ea2b1d47a619c8c066877843cd8a7745ce31021a8d1548cf2a"],"legacy_cameraID":1,"type":"ip","source":"google","country":"JP","city":"Tokyo","is_active_image":false,"is_active_video":false,"utc_offset":32400,"timezone_id":"Japan Standard Time","timezone_name":"Japan Standard Time","reference_url":"101.110.193.152/","retrieval":{"ip":"101.110.193.152","port":"80","video_path":"/"},"__v":0}

I also have a list of camera IDs that I want to remove from the original file in the format:

5b182800751c3b00044514a9
5b1976b473569e00045dba59
5b197b1273569e00045ddf0f
5b1970cc73569e00045d94fc

How can I use grep or some other command line utility to remove all lines in the input file that have an ID listed in the second file?

Answer 1

Let's say that you have a file called ids.txt that has all of the camera id's that need to be excluded from your data file, which we'll call data.json . We can use the -f option of grep (match from a file) and the -v option (only output non-matching lines) as follows:

grep -f ids.txt -v data.json 

grep will only output lines of data.json that do not match any lines in ids.txt .

Answer 2

You should use json aware tool. Here is a GNU awk script that uses json extension:

$ gawk '                                     # GNU awk
@load "json"                                 # load extension
NR==FNR {                                    # read oids to a hash
    oid[$0]
    next
}
{                                            # process json
    lines=lines $0                           # support multiline json form
    if(json_fromJSON(lines,data)!=0) {       # once json is complete
        if(!(data["_id"]["$oid"] in oid))    # test if oid in exclude list
            print                            # output if not
        lines=""                             # rinse for repeat
    }
}' oids json

Answer 3

A simple thing you can do is get ids from camera info and check if they are listed in the second file.

For example:

#!/bin/bash
exec 3<info.txt
while IFS= read -r line <&3; do
  id="$(printf '%s' "${line}" | jq '._id."$oid"' | sed -e 's/"//g')"
  if ! grep -e "${id}" list.txt >/dev/null; then
    printf '%s\n' "${line}"
  fi
done >clean.txt
exec 3>&-

Where:

1. info.txt is the file with camera information
2. list.txt is the list of ids you do not want

Note that this is not the only way you can achieve it, I used a simple cycle just as poc.

You can achieve it using directly jq, for example:

#!/bin/bash
for id in $(jq '._id."$oid"' info.txt | sed -e 's/"//g'); do
  if ! grep -e "${id}" list.txt >/dev/null; then
    grep -e "${id}" info.txt
  fi
done >clean.txt

Note that in this second example the second grep is needed because you never take the whole line of the into.txt file, only the id.

Also, be aware that if you have an alias like alias grep='grep --color=always' it could break your output.

Answer 4

假设您的 json 文件始终是常规的：

awk -F'"' 'NR==FNR{ids[$1]; next} !($6 in ids)' ids json