awk正则表达式：使用它或不使用变量之间的区别

Question

I have an awk script that behaves different when I put a regular expression in different places. Obviously I make the logic of the program to work the same in both cases, but it does not. The script is for analyzing some logs where each transaction has an unique ID. The log looks like

timestamp (ID) more info

for example:

2014-10-06 05:24:40,035 INFO  (4aaaaaaaaabbbbbbcccb) [somestring] body with real information and a key string that determines the type of thransaction
2014-10-06 05:24:40,035 INFO  (4aaaaaaaaabbbbbbcccb) [somestring] body with other information
2014-10-06 05:24:40,035 INFO  (4aaaaaaaaabbbbbbcccb) [somestring] body with more information
2014-10-06 05:24:40,035 INFO  (4xxbbbbbbbbbbbbbcccb) [somestring] this is a different transaction

What I want is to process all the log lines of a certain type of transaction to see how much time do they take. Each transaction is spread across several log lines and its identified by its unique ID. To know if a certain transaction is of the type I want I have to search for certain string in the first line of that transaction . In the log could be lines without the above format.

What do I want:

• Distinguish if the current line is part of a transaction (it has an ID)
• Check if the ID is already registered in an cumulative array.
  • If not, check if it is of the desired type: search for a fixed string in the body of the line.
  • If it is, register the timestamp, and blah blah

And here is the code (note this is a very minified version).

This is what I would like to use, first check if it is a transaction line and after check if it is of the correct type

awk '$4 ~ /^\([:alnum:]/
{
  name=$4;gsub(/[()]|:.*/,"",name);++matched
  if(!(name in arr)){
    if($0 ~ /transaction type/){arr[name]=1;print name}}
}END
{
  print "Found :"length(arr)
  print "Processed "NR
  print matched" lines matched the filter"
}'

That script only finds 868 transactions and there are some more than 14K. If I change the script to look like the code below if finds all the 14k transactions, but only the first line of all of them, so it is not useful for me.

awk '/transaction type/
{
  name=$4;gsub(/[()]|:.*/,"",name);++matched
  if(!(name in arr)){
    arr[name]=1;print name
   }
}END
{
  print "Found :"length(arr)
  print "Processed "NR
  print matched" lines matched the filter"
}'

Thanks in advance.

Edit

Shame on me. There were more than one actual problem in this topic. The main one was that the regex was not matching the proper string. The ID string and the type of transaction string were on the same line, that is true, but on those lines the ID was like (aaaaaabbbbbcccc: ), with two spaces at the end. That makes AWK to parse "(aaaaaaaabbbbcccc:" and ")" as two different fields. I realized when I did

$4 !~ /regex/ print $4

and a lot of valid IDs appeared.

The second problem, which appeared after fixing the regular expression have been addressed by some people here. Having the main regular expression and the firs { in separated lines makes awk to print each record. I realized that myself and the same day later I read here the solutions. Amazing.

Thank you very much to every one. I can only accept one answer as valid, but I learned a lot from all of them.

Answer 1

It's only a syntax error. When you use a posix character class you must enclose it between square brackets:

[[:alnum:]]

Otherwise [:alnum:] is seen as a character class that contains : almnu

Answer 2

white space matters in awk. This:

/foo/ {
    print "found"
}

means print 'found' every time "foo" is present while this:

/foo/
{
    print "found"
}

means print the current record every time "foo" is present and print "found" for every single input record so chances are when you wrote:

$4 ~ /^\([:alnum:]/
{
     ....
}

you actually meant to write:

$4 ~ /^\([:alnum:]/ {
     ....
}

also, chances are you meant to use the POSIX character class [[:alnum:]] instead of the set of characters [ : alnum as described by the character set [:alnum:] :

$4 ~ /^\([[:alnum:]]/ {
     ....
}

If you fix those things and you still need help, provide some testable sample input and expected output we can help you more.

Answer 3

So in brief if I understood properly you wish to get ids of certain type of transactions.

First assumption: id and transaction type are on the same line, something like this should do (largely adapted from your code)

awk 'BEGIN {
  matched=0 # more for clarity than really needed
}
/\([[:alnum:]]*\).*transaction type/ { # get lines matching the id and the transaction only
  gsub(/[()]/,"",$4) # strip the () around the id
  ++matched # to get the number of matched lines including the multiples ones.
  if (!($4 in arr)) { # as yours, if the id is not in array
    arr[$4]=1 # add the found id to array for no including it twice
    print $4 # print the found id (only once as we're in the if
  }
}
END { # nothing changed here, printing the stats...
  print "Found :"length(arr)
  print "Processed "NR
  print matched" lines matched the filter"
}'

Output of this from your sample input:

prompt=> awk 'BEGIN { matched=0}; / \([a-z0-9]*\) / { gsub(/[()]/,"",$4); ++matched; if (!($4 in arr)) { arr[$4]=1; print $4 }}; END { print "Found: "length(arr)"\nProcessed "NR"\n"matched" lines matched the filter" }' awkinput
4aaaaaaaaabbbbbbcccb
4xxbbbbbbbbbbbbbcccb
Found: 2
Processed 4
4 lines matched the filter

I've ommitted the transaction in the test as I've no clue on what it may be