Shell Script loop is executing multiple times

Question

I have a log file. I'm doing tail -f and grep options whenever new logs are coming. I'm facing loop issue, It is executing multiple times. here is my script,

AuditTypeID=$""
QueryResult=$""
tail -n 0 -F hive-server2.log \
 | while read LINE
 do
   if [ `echo $LINE | grep -c "select *" ` -gt 0 ]
    then
      AuditTypeID=15
      QueryResult=$(
       awk '
          BEGIN{ print "" }
          /Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 }
          /s3:\/\//{ print "," q }
          ' OFS=',' hive-server2.log \
       | sed -n \$p
       )
    elif [ `echo $LINE | grep -c 'select count' ` -gt 0 ]
     then
       AuditTypeID=22
       QueryResult="$(
         grep -oE 'select count\(.\) from [a-zA-Z][a-zA-Z0-9]*' hive-server2.log \
          | sed -n \$p
         )"
     fi

    user=$(
       cat hive-server2.log \
        | grep user \
        | awk -F "[. ]" '{print "," $(NF-1)}' \
        | tr -d ',' \
        | tr -d 'UTC'
       )
     Additional_Info=$(
        echo -e "{\"user\":\"""${user}""\", \"query\":\"""${QueryResult}""\",\"s3Path\":\"""${s3}""\"}"
        )
    echo -e "$Additional_Info" > op.json
    for file in /var/log/hive/op.json
     do
       boto-rsync $file s3://hive-log/log/script/$file.$current_time
     done
 done

It will filter the operations based on the keyword. For some reason it is executing multiple times. I need to save the output for only one instance and any help to simplify the logic is appreciated.

Answer 1

First thing I see in your script is that in the first awk scriptlet inside the if statement you seem to be reparsing the whole of hive-server2.log (which is probably racy/bad because you are tailing to your script, and hive-server.log is growing?)... and this reparsing of the log seems to be a common theme in the script -- I think this is the root cause of the issue.

One simplification ;) readily apparent is removal of the elif code -- it will never run because /select count/ will be matched by the if statement's /select */ .

To truly take a stab at simplifying this, my strategy would be to rewrite the whole of this in awk . There is nothing that you are doing here that is beyond awk 's built-in capabilities -- and awk can fire off external shell commands as easily as sh . The awk implementation will also likely be much faster.

I started trying to do this translation, but with the way you are specifying multiple reparsing of hive-server2.log , I frankly got lost. Having a bit of input and intended output would help here... Please post hive-server2.log and your expected output.