read data from a file and insert into a database using bash or awk

Question

I'm very new in Bash and unix. I was assigned a task to extract data from a log called errors.out which uses the delimiter | . Here's an example of the data:

1423544401|video_load_time|5.0.0.37|i3|55.66.88.77|0.0|RAT|mike|5656519|6549943438483||album|||N/A||7149

so basically the data in order is delimited by | is the following:

   times
   eventKey
   appl
   devic
   devn
   sign
   ne
   State
   latude
   lde
   su
   buame
   tount
   assId
   eCode
   monitor
   duration

I need to count each of the eventkeys (video_load_time) and add their total duration. Then I have to insert those two fields into a database which i have created. My database has 2 fields count and totalduration under the table counter . Does anyone has any ideas what I should use to accomplish this? I heard awk or python would be a good choice..

Answer 1

With awk:

awk -F \| '{ duration[$2] += $NF } END { for(d in duration) { print d, duration[d] } }' errors.out

Here $2 is the second field in each line and $NF the last, so this sums up the durations by key and prints the results when the end of the input is reached -- after everything is processed.

EDIT: To add a counter to that,

    awk -F \| '{ duration[$2] += $NF; ++counter[$2] } END { for(d in duration) { print d, counter[d], duration[d] } }' errors.out

Answer 2

awk -v FS="|" -v KEY="video_load_time" '$2==KEY{TOTAL+=$NF}END{print KEY,TOTAL}'  my_file

for

-v KEY="video_load_time"

you can change the variable key string to any other value other than video_load_time