Parsing text file lines

Question

I have a log file that contains, among other data, lines like these:

2012-05-23T20:52:11+00:00 heroku[router]: GET myapp.com/practitioner_activities/10471/edit dyno=web.2 queue=0 wait=0ms service=866ms status=200 bytes=48799
2012-05-23T20:52:46+00:00 heroku[router]: GET myapp.com/users/sign_out dyno=web.1 queue=0 wait=0ms service=20ms status=302 bytes=88
2012-05-23T20:52:46+00:00 heroku[router]: GET myapp.com/ dyno=web.13 queue=0 wait=0ms service=18ms status=200 bytes=4680
2012-05-23T20:53:04+00:00 heroku[router]: POST myapp.com/p/ENaCXExu7qNEqzwYYyPs dyno=web.5 queue=0 wait=0ms service=207ms status=302 bytes=119
2012-05-23T20:53:04+00:00 heroku[router]: GET myapp.com/practitioner_activities/welcome dyno=web.3 queue=0 wait=0ms service=57ms status=200 bytes=5061
2012-05-23T20:53:04+00:00 heroku[router]: GET myapp.com/assets/application-print-715276cc0b76d0d82db3ab5866f22a23.css dyno=web.14 queue=0 wait=0ms service=9ms status=200 bytes=76386

I'd like to parse them and dump them into a file I can open with excel for analysis. I need the Hour, Minute, Verb (GET or POST), url and the 'service=' time.

For example, for the first line above:

2012-05-23T20:52:11+00:00 heroku[router]: GET myapp.com/practitioner_activities/10471/edit dyno=web.2 queue=0 wait=0ms service=866ms status=200 bytes=48799

I'd expect the output to look something like:

"20", "52", "GET", "myapp.com/practitioner_activities/10471/edit", "866"

How would I do this in awk or with a short ruby script?

Answer 1

Using awk , you could try something like:

awk '{ OFS="\", \""; split ($8, array, "="); printf "\"" substr ($1 , length ($1) - 13, 2 ) OFS substr ($1 , length ($1) - 10, 2 ) OFS $3 OFS $4 OFS substr (array[2], 0, length (array[2]) -2) "\"\n" }' file.txt

Results:

"20", "52", "GET", "myapp.com/practitioner_activities/10471/edit", "866"
"20", "52", "GET", "myapp.com/users/sign_out", "20"
"20", "52", "GET", "myapp.com/", "18"
"20", "53", "POST", "myapp.com/p/ENaCXExu7qNEqzwYYyPs", "207"
"20", "53", "GET", "myapp.com/practitioner_activities/welcome", "57"
"20", "53", "GET", "myapp.com/assets/application-print-715276cc0b76d0d82db3ab5866f22a23.css", "9"

HTH

Edit:

awk '{ OFS="\", \""; ORS="\"\n"; split ($8, array, "="); print "\"" substr ($1 , 12, 2 ), substr ($1 , 15, 2 ), $3, $4, array[2] + 0 }' file.txt

Thanks Dennis! The code is much much nice now :-)

Answer 2

A ruby answer

ruby -ane '
    hr, min = $F[0][/(?<=T)\d\d:\d\d/].split(/:/)
    svc = $F[7].split(/=/)[-1]; svc[/ms/] = ""
    puts %q{"%d", "%d", "%s", "%s", "%d"} % [hr, min, $F[2], $F[3], svc]
' logfile