Bash：如何从txt文件读取行，如何按列值识别和删除重复项

Question

I'm writing a Bash script, in which I'm collecting data from some logfiles, removing the duplicate values, sort it by username and storing the output in the output.txt. What I would like to do next, is to read the output.txt line by line and whenever the username appears more than once, to create a new line which will have data from both lines.

The purpose of this, is to send an email to the user ONLY ONCE and inform him that he can't use this feature on this server.

I don't know if I'm explaining it well.. For example see below the output.txt

output.txt

13:49:19 DENIED: "Software_1" UserA serv7 (Can't run this feature. )
13:49:19 DENIED: "Software_2" UserA serv7 (Can't run this feature. )
15:09:14 DENIED: "Software_3" UserB serv5 (Can't run this feature. )
15:09:15 DENIED: "Software_4" UserB serv5 (Can't run this feature. )
17:20:43 DENIED: "Software_3" UserC serv5 (Can't run this feature. )
17:20:43 DENIED: "Software_5" UserC serv8 (Can't run this feature. )

expected result

Software_1, Software_2, UserA serv7 (Can't run this feature. )
Software_3, Software_4, UserB serv5 (Can't run this feature. )
Software_3, Software_5, UserC serv5, serv8 (Can't run this feature. )

Can someone suggest a solution and explain how it's working?

Answer 1

process.awk:

#! /usr/bin/awk -f
NF {
     sft[$4] = sft[$4] $3 ", ";
     if (!(srv[$4] ~ $5))
         srv[$4] = srv[$4] $5 ", ";
     mesg[$4] = $6 " " $7 " " $8 " " $9 " " $10
}

END {
    for (user in sft) {
        gsub("\"", "", sft[user]);
        print  sft[user], user, srv[user], mesg[user];
    }
}

This AWK script checks if a line is not blank (using the number of fields built-in variable, NF). If so, it maintains an array for software and server, indexing them by the user field. For each user type, it appends the server and software associated to it. This happens for each line.

When all the lines in the input file is done processing, in the END pattern, it iterates over all the entries in the software array printing the associated software, users, servers and the message.

To run it

awk -f process.awk output.txt

Answer 2

$ cat userlog_unparsed.log
13:49:19 DENIED: "Software_1" UserA serv7 (Can't run this feature. )
13:49:19 DENIED: "Software_2" UserA serv7 (Can't run this feature. )
15:09:14 DENIED: "Software_3" UserB serv5 (Can't run this feature. )
15:09:15 DENIED: "Software_4" UserB serv5 (Can't run this feature. )
17:20:43 DENIED: "Software_3" UserC serv5 (Can't run this feature. )
17:20:43 DENIED: "Software_5" UserC serv8 (Can't run this feature. )


$ awk '
     { sws[$4][$3]++; srvs[$4][$5]++; }
     END{
         for(user in sws){
             swuser="";srvuser="";
             for(sw in sws[user]){swuser=swuser","sw}
             for(srv in srvs[user]){srvuser=srvuser","srv};
             print substr(swuser,2) ", " user ", " substr(srvuser,2);
         }
     }' userlog_unparsed.log

"Software_2","Software_1", UserA, serv7
"Software_3","Software_4", UserB, serv5
"Software_3","Software_5", UserC, serv5,serv8

Explanation:

1. Record all the users & their softwares, servers.
2. At the end, loop through all of them, & append the users, their respective servers & softwares. & print them.