使用bash和awk将日志文件分组

Question

I'm trying to sort a log file in a specific way, but I'm not sure how to perform the last step.

My logfile has entries like this:

Feb 15 17:00:34 server sshd[13879]: Invalid user test from 200.242.94.133
Feb 15 17:00:35 server sshd[13780]: Invalid user ftpuser from 200.242.94.133
Feb 15 17:01:34 server sshd[13890]: Invalid user test from 200.242.94.133
Feb 15 17:01:35 server sshd[13791]: Invalid user vnc from 200.242.94.133
Feb 15 17:01:35 server sshd[13794]: Invalid user test from 50.63.172.108
Feb 15 17:01:36 server sshd[13798]: Invalid user vnc from 50.63.172.108

I use the command:

cat logfile | grep "Invalid user" | awk '{print $8 ", " $10 }' | sort -t":" -k2,2 | uniq -c

Which outputs:

 1 ftpuser, 200.242.94.133
 2 test, 200.242.94.133
 1 test, 50.63.172.108 
 1 vnc, 200.242.94.133
 1 vnc, 50.63.172.108

I'd like to get:

1 ftpuser, (1) 200.242.94.133
3 test, (2) 200.242.94.133, (1) 50.63.172.108
2 vnc, (1) 200.242.94.133, (1) 50.63.172.108

I'm not sure how to sum the words column while keeping the ip address counted separate and then including it with other results.

Attempt with answer:

# awk '/Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}' logfile | sort -k2
awk: /Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}
awk:                                 ^ syntax error
awk: /Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}
awk:                                                                                                   ^ syntax error
awk: /Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}
awk:                                                                                                                               ^ syntax error

Answer 1

$ awk '/Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}' logfile
2 vnc, (1) 50.63.172.108, (1) 200.242.94.133
1 ftpuser, (1) 200.242.94.133
3 test, (1) 50.63.172.108, (2) 200.242.94.133

If you want it sorted alphabetically by user:

$ awk '/Invalid user/{user[$8]++;ip[$8][$10]++} END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}' logfile | sort -k2
1 ftpuser, (1) 200.242.94.133
3 test, (1) 50.63.172.108, (2) 200.242.94.133
2 vnc, (1) 50.63.172.108, (1) 200.242.94.133

The above works with GNU awk . I haven't tested with BSD.

How it works

• /Invalid user/{user[$8]++;ip[$8][$10]++}

  For any line in logfile with an invalid user, this counts up the user name, field 8, and the ip address, field 10.

• END{for (u in user){printf "%s %s",user[u],u;for (i in ip[u])printf ", (%s) %s",ip[u][i],i;print""}}

  When we have finished reading logfile , this loops through every user that we have seen and prints the number of times that we have seen that user, followed by that user's name, followed by, for each ip address, a count for that ip followed by that ip.

Answer 2

John1024's answer is a very concise and presumably fast solution that is an option, IF:

• you're using GNU awk (the solution uses non-POSIX features that won't work with BSD awk (also used on OS X) or mawk, for instance).
• you don't mind a seemingly random order of IP addresses (due to unsorted key enumeration of an associative array; however, in GNU awk 4.0+, you can use PROCINFO["sorted_in"] to control the enumeration order ).

Here is a much more pedestrian solution, which, however:

• uses only POSIX awk features.
• lists IP addresses in the order encountered in the input.

It builds on a slightly simplified version of the OP's command.

awk '/Invalid user/ { print $8 ", " $10 }' logfile | sort -t":" -k2,2 | uniq -c |
awk '
    # Helper output function for printing an output line.
  function printLine(c, n, l) { 
    sub(/,$/, "", n); print c, n l
  }
    # End of previous block found (new username)?
  prevName != $2 {
      # Print summary line for previous block.
    if (NR>1) printLine(count, prevName, ipList)
      # Start new block.
    prevName=$2; count=0; ipList=""
  }
    # Build block summary values.
  { 
    count+=$1
    ipList=ipList ", (" $1 ") " $3
  }
    # Print summary line for last block.
  END { 
    printLine(count, prevName, ipList)
  }
  '