Comparing txt file to third column in a csv bash

Question

I am very new to programming and decided to learn bash as we deal with some log servers that are Linux/Unix based and so scripting is a bit easier.

I have a cvs file that is laid out as follows:

PC,user,file,path - all comma separated.

I have a white list of file names that are line separated. Some include spaces.

My goal is to compare the whitelist to column 3 of the csv file and output all lines that don't match. I have tried a while read loop with an if statement but cannot seem to get it to work. I have done a few awk one liners and actually got one from a past stackoverflow post that outputted the lines that matched the whitelist but I cannot seem to figure out how to reverse to the logic to get it to work. Code is below.

awk     'BEGIN{i=0}
       FNR==NR { a[i++]=$1; next }
        { for(j=0; j<i; j++)
            if(index($0,a[j]))
                {print $0;break}
        }' $whitelist $exestartup

I would like to stick to basic bash with no add-ons and not opposed to doing a loop/if statement instead of an awk one liner.

Sample input/output:

whitelist.txt

program.exe
super program.exe
possible-program.exe

exestartup.csv

Asset1,user1,potato.exe,c:\\users\\user1
Asset2,user2,program.exe,c:\\users\\user2
Asset3,user3,possible-program.exe,c:\\users\\user3
Asset4,user4,super program.exe,c:\\users\\user4

Output

Asset1,user1,potato.exe,c:\\users\\user1

Answer 1

awk to the rescue!

awk -F, 'FNR==NR{a[$1]; next} !($3 in a)' whitelist exestartup

set the field delimiter to comma. Load all whitelist names and compare against $3 fields of the file, if not match; print.

If you post sample input and expected output you'll get more answers and perhaps better suggestions.

using your input files

$ awk -F, 'FNR==NR{a[$1]; next} !($3 in a)' whitelist.txt exestartup.csv

Asset1,user1,potato.exe,c:\users\user1

if your awk is broken and the field values are disjoint you can revert to grep

$ grep -vf whitelist.txt exestartup.csv

Asset1,user1,potato.exe,c:\users\user1

Answer 2

Using join :

$ join -v 1 -t, -1 3 -2 1 -o 1.1,1.2,1.3,1.4 <(sort -t, -k3,3 exestartup.csv) <(sort whitelist.txt)
Asset1,user1,potato.exe,c:\users\user1

If the input files are already sorted on the matching key (they don't appear to be in your example), that could simply be:

$ join -v 1 -t, -1 3 -2 1 -o 1.1,1.2,1.3,1.4 exestartup.csv whitelist.txt

Answer 3

This solution uses only Bash 3 builtins:

IFS=$'\n' read -d '' -r -a whitefiles < whitelist.txt

while IFS= read -r csvline || [[ -n $csvline ]] ; do
    IFS=, read pc user file path <<< "$csvline"
    for wfile in "${whitefiles[@]}" ; do
        [[ $wfile == "$file" ]] && continue 2
    done
    printf '%s\n' "$csvline"
done < exestartup.csv

A much faster and cleaner solution can be implemented in Bash 4 because it's got associative arrays.