How do i parse the large file in linux

Question

I am beginner for Linux. I have the following flat file test.txt

Iteration 1   
Telephony   


Pass/Fail

5.1.1.1   voiceCallPhoneBook   50   45
5.1.1.4   voiceCallPhoneHistory   50   49
5.1.1.7   receiveCall   100   100
5.1.1.8   deleteContacts   20   19
5.1.1.9   addContacts   20   20
Telephony   16:47:42
Messaging   


Pass/Fail

5.1.2.3   openSMS   50   49
5.1.2.1   smsManuallyEntryOption   50   50
5.1.2.2   smsSelectContactsOption   50   50
Messaging   03:26:31
Email   


Pass/Fail

Email   00:00:48
Email   


Pass/Fail

Email   00:00:40
PIM   


Pass/Fail

5.1.6.1   addAppointment   5   0
5.1.6.2   setAlarm   1   0
5.1.6.3   deleteAppointment   5   0
5.1.6.4   deleteAlarm   1   0
5.1.6.5   addTask   1   0
5.1.6.6   openTask   1   0
5.1.6.7   deleteTask   1   0
PIM   00:03:06
Multi-Media   

teration 2   
Telephony   


Pass/Fail

5.1.1.1   voiceCallPhoneBook   50   47
5.1.1.4   voiceCallPhoneHistory   50   50
5.1.1.7   receiveCall   100   100
5.1.1.8   deleteContacts   20   20
5.1.1.9   addContacts   20   20
Telephony   04:02:05
Messaging   


Pass/Fail

5.1.2.3   openSMS   50   50
5.1.2.1   smsManuallyEntryOption   50   50
5.1.2.2   smsSelectContactsOption   50   50
Messaging   03:20:01
Email   


Pass/Fail

Email   00:00:47
Email   


Pass/Fail

Email   00:00:40
PIM   


Pass/Fail

5.1.6.1   addAppointment   5   5
5.1.6.2   setAlarm   1   1
5.1.6.3   deleteAppointment   5   5
5.1.6.4   deleteAlarm   1   1
5.1.6.5   addTask   1   1
5.1.6.6   openTask   1   1
5.1.6.7   deleteTask   1   1
PIM   00:09:20
Multi-Media   

I want to count the number of occurrences for specific word in the file Eg: if i search with "voiceCallPhoneBook" it's display as 2 times.

i can use

cat reports.txt | grep "5.1.1.4" | cut -d' ' -f1,4,7,10 |

after running this script i got output like below

5.1.1.4 voiceCallPhoneBook  50  45
5.1.1.4 voiceCallPhoneBook  50  47

It is very large file and i want to make use of loops with bash/awk scripts and also find the average of SUM of 3rd and 4th column value. i am struggling to write in bash scripts. It would be appreciated someone can give the solution for it.

Thanks

Answer 1

#!/usr/bin/awk -f
BEGIN{
c3 = 0
c4 = 0
count = 0
}

/voiceCallPhoneBook/{
    c3 = c3 + $3;
    c4 = c4 + $4;
    count++;
}

END{
 print "column 3 avg: " c3/count
 print "column 4 avg: " c4/count
}

1) save it in a file for example countVoiceCall.awk

2) awk -f countVoiceCall.awk sample.txt

output:

column 3 avg: 50
column 4 avg: 46

Briefly explain:

a.    BEGIN{...} block uses for variables initialization
b.    /PATTERN/{...}  blocks uses to search your keyword, for example "voiceCallPhoneBook"
c.    END{...} block uses for print the results

Answer 2

This will search for lines containing 5.1.1.4
Make a tally of the 3rd and 4th columns
Then print them all out

awk '/^5\.?\.?\.?/ {a[$1" " $2] +=$3 ; b[$1" " $2] +=$4 }
END{ for (k in a){
      printf("%-50s%-10i%-10i\n",k,a[k],b[k])}
}' $1

Duplicate from earlier today is here Parse the large test files using awk

With headers avg and Occurence count and formatted a bit neater for easier reading :)

awk 'BEGIN{
            printf("%-50s%-10s%-10s%-10s\n","Name","Col3 Tot","Col4 Tot","Ocurr")
}

/^5\.?\.?\.?/ {
        count++
        c3 = c3 + $3
        c4 = c4 + $4
        a[$1" " $2] +=$3
        b[$1" " $2] +=$4
        c[$1" " $2]++
}

END{
        for (k in a)
                {printf("%-50s%-10i%-10i%-10i\n",k,a[k],b[k],c[k])}

                print "col3 avg: " c3/count "\ncol4 avg: " c4/count
}' $1