使用 awk 和 sed 进行日志解析

Question

I have

2019-11-14T09:42:14.150Z  INFO ActivityEventRecovery-1 ActivityCacheManager - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Handling activity 0082bc26-70a6-433e-a470-
2019-11-14T09:43:08.097Z  INFO L2HostConfigTaskExecutor2 TransportNodeAsyncServiceImpl - FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Calling uplinkTeamingChangeListener.onTransportNodeUpdated on TN 72f73c66-da37-11e9-8d68-005056bce6a5 revision 5
2019-11-14T09:43:08.104Z  INFO L2HostConfigTaskExecutor2 Publisher - ROUTING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Refresh mac address of Logical router port connected with VLAN LS for logical router LogicalRouter/f672164b-40cf-461f-9c8d-66fe1e7f8c19
2019-11-14T09:43:08.105Z  INFO L2HostConfigTaskExecutor2 GlobalActivityRepository - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Submitted activity 73e7a942-73d2-4967-85fa-7d9d6cc6042b in QUEUED state with dependency null exclusivity true and requestId null and time taken by dao.create is 1 ms

these kind of logs where I want to parse it into a json object. Till now, I was using python regex and putting it into a dictionary.

    currentDict = {
                               "@timestamp" : regexp.group(1),
                               "Severity" : regexp.group(2),
                               "Thread" : regexp.group(3),
                               "Class" : regexp.group(4),
                               "Message-id" : regexp.group(5),
                               "Component" : regexp.group(6),
                               "Message" : regexp.group(7),
                               "id's" : re.findall(x[1], regexp.group(7))
                        }

but this way it is very slow ie it is taking 5-10 mins for 200mb file.

Python regex I used - (\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d.\d\d\dZ)\s+(INFO|WARN|DEBUG|ERROR|FATAL|TRACE)\s+(.*?)\s+(.*?)\s+\-\s+(.*?)\s+(?:(\[?.*?\])?)\s(.*)

Expected output -

{"@timestamp" : "2019-11-14T09:42:14.150Z", "Sevirity" : "INFO", "Thread" : "ActivityEventRecovery-1", "Class" : "ActivityCacheManager - -", "Component" : "[nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"]", "Message" : "Handling activity 0082bc26-70a6-433e-a470-"}
{"@timestamp" : "2019-11-14T09:43:08.097Z", "Sevirity" : "INFO", "Thread" : "L2HostConfigTaskExecutor2", "Class" : "TransportNodeAsyncServiceImpl - FABRIC", "Component" : "[nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"]", "Message" : "Calling uplinkTeamingChangeListener.onTransportNodeUpdated on TN 72f73c66-da37-11e9-8d68-005056bce6a5 revision 5}"}
{"@timestamp" : "2019-11-14T09:43:08.104Z", "Sevirity" : "INFO", Thread : "L2HostConfigTaskExecutor2", "Class" : "Publisher - ROUTING", "Component" : "[nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"]", Message : "Refresh mac address of Logical router port connected with VLAN LS for logical router LogicalRouter/f672164b-40cf-461f-9c8d-66fe1e7f8c19}"}
{"@timestamp" : "2019-11-14T09:43:08.105Z", "Sevirity" : "INFO", "Thread" :  "L2HostConfigTaskExecutor2", "Class" :   "GlobalActivityRepository", "Component" : "[nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"]", "Messages" : "Submitted activity 73e7a942-73d2-4967-85fa-7d9d6cc6042b in QUEUED state with dependency null exclusivity true and requestId null and time taken by dao.create is 1 ms"}}

On internet, I found out that using awk and sed it can be done faster. I don't know much about it. How to do parsing using awk and sed .

Please Help!

Answer 1

# For timestamp
cut -d " " -f 1 in > temp
sed -i -e 's/^/{"@timestamp" : "/' temp
awk 'NF{print $0 "\", "}' temp > a

# For Severity ...

# For Thread ...

# For Class
cut -d " " -f 5,6,7 in > temp
sed -i -e 's/^/"Class" : "/' temp
awk 'NF{print $0 "\", "}' temp > d

# For Component
grep -o -P '(?<=\[).*(?=\])' in > temp
sed -i -e 's/^/"Component" : \["/' temp
awk 'NF{print $0 "\"], "}' temp > e

# For Message ...

# Merge all files line by line
paste -d " " a b c d e f

I'll explain some of this script in short, cut is used to get the word in between two spaces. Sed is adding the text to the beginning of each line. Awk is adding text to the end of each line.

I have left the Severity, Thread and Message section as they are same as the other ones. The script is fairly simple, but you won't understand it without knowing how to use the tools itself, since you said you don't know about them.

Answer 2

This sed script should do a very fast job:

sed -E 's/^([^ ]+)\s+([^ ]+)\s+([^ ]+)\s+([^ ]+\s-\s[^ ]+)\s+(\[.*\])\s+(.*)$/{"@timestamp" : "\1", "Severity" : "\2", "Thread" : "\3", "Class" : "\4", "Component" : "\5", "Message" "\6"}/' inputdata.dat

Explanation:

• sed 's/^<regular-expression>$/<output-string>/' manipulates/substitutes every ^<input-line>$ , that matches the regex. Hereby ^ means begin and $ end of line .

• -E : means use extended regular expressions . Extended regular expressions are now included in POSIX. In grep's manual page can you find:

  Basic vs Extended Regular Expressions:

  In basic regular expressions the meta-characters?, +, {, |, (, and ) lose their special meaning; instead use the backslashed versions \?, \+, \{, \|, \(, and \).

• (...)...(...)...(...)... : the content inside (...) is - in this script - a description to match the first, second, third... sixth data field. In general, it's an instrument to subdivide the whole regular expression into different units that you can refer to in the output string via \1 , \2 ... . Doing so, you can can limit text manipulations to certain contexts. In this case the data fields themselves stay unchanged, only the context shall change.

• [^ ]+ : within [...] a class of characters is described [A-Za-z0-9] means exactly one char of alphabet or digits , [0-9]+ means at least one digit , [ ] means one blank , [^0-9 ] means exactly one char - anything else but digit and but blank , so here [^ ]+ means at least one char - anything else but blank -> the right regex pattern for the content of the first three data fields
• ([^ ]+\s-\s[^ ]+) : the fourth data field "Class" is a compound data field, two components and a separator ' - ' (1st - 2nd), outside of char classes ( [..] ) use \s instead of ' '
• (\[.*\]) : the fifth data field "Component" is also a compound data field but enclosed in square brackets [ and ] . In order to match the bracket-character itself (not a char class by enclosing those characters in brackets) you have to use the bracket-character [ or ] preceded by a backslash \ . . is a wildcard, so .* in \[.*\] means everything between the brackets
• \s+ : at least one blank or tabulator (between the data fields)
• so the sixth data field - the message of flexible length - can be matched as (.*) what means the rest after ] and the directly following whitespace till end of line
• \1... \6 (in the right part): the references in the output sting to the respecive expression group (in this case the data fields) within the regular expression.
• inputdata.dat : replace this by the name of your data file

In order to get a runnable shell script save this as a file:

#! /bin/sh
sed -E 's/^([^ ]+)\s+([^ ]+)\s+([^ ]+)\s+([^ ]+\s-\s[^ ]+)\s+(\[.*\])\s+(.*)$/{"@timestamp" : "\1", "Severity" : "\2", "Thread" : "\3", "Class" : "\4", "Component" : "\5", "Message" "\6"}/' "$1" >"$2"

After that run command chmod +x <your-scriptname> to make the script executable. Then it can be run as ./<your-scriptname> <input-file> <wanted-output-file-name> .

Attention:

• Do NOT use the same filename for input and output file
• If a file with the output filename already exists, it will be overridden.