shell Script NO lines are append from input file using time stamp

Question

INPUT-file is /var/log/error.log

 Apr 7 05:58:05 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical     
 Apr 7 09:21:12 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical     
 Apr 7 13:05:57 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical

My Script

#!/bin/ksh

d1=$(date --date="-50 min" "+%b %_d %H:%M")

d2=$(date "+%b %_d %H:%M")

while read line; do

    [[ $line > $d1 && $line < $d2 || $line =~ $d2 ]] && echo $line | egrep -wi '[Cc]ritical' >> /mnt/dinesh/ksh-scripts/draft/text.txt

done < /var/log/error.log

While running my script the system TIME is around 13:06:05

So as per the script the /mnt/dinesh/ksh-scripts/draft/text.txt FILE should have the last entry of my error.log But NO lines are appened in mnt/dinesh/ksh-scripts/draft/text.txt .

Can someone help me to fix this issue.

Answer 1

The essential problem with the logic of the code is that is using string comparisons to compare dates. In a string comparison, Apr 17 is less than Apr 2 and Feb 1 is less than Jan 1 .

Instead, the pattern in POSIX shells, is to turn the date into seconds since 1970-01-01 00:00:00 UTC, also known as the "epoch".

For the date command the way to give time in this seconds format to set FORMAT to +%s , eg date +%s .

In ksh there is a printf format specifier, T , to turn a string such as a line read in from your log file into the number of second since the epoch. See https://www.unix.com/302701867-post4.html?s=860f6c21431fa69ef9e083161f93739d

Here is your script rewritten to address the logic bug by converting comparisons to seconds:

#!/bin/ksh
ERROR_LOG=/tmp/error.log
FILTERED_ERROR_LOG=/tmp/text

now_secs=$(date "+%s")
start_secs=$(date --date="-30000 min" +%s)

while read line; do
    # Below stderr is redirected since I can't figure out how to get just the first part
    # of the date parsed skipping the other stuff after that.
    # Even though there is warning printed, the time comes out parsed correctly
    line_secs=$(printf "%(%#)T" "$line" 2>/dev/null)

    if (( $line_secs > $start_secs && line_secs < now_secs || line_secs == now_secs )); then
        echo $line | egrep -wi '[Cc]ritical' >> $FILTERED_ERROR_LOG
    fi
done < $ERROR_LOG

Some additional notes.

It appears like you more broadly may want to understand such debug such problems. As you can see, asking on StackOverflow such a very specific kind of problem is probably not going to be of much interest to others so it may take a while to get any sort of meaningful response.

The basic trace option is -x . Let's run that on the above script:

ksh -x /tmp/bug.ksh
+ ERROR_LOG=/tmp/error.log
+ FILTERED_ERROR_LOG=/tmp/text
+ date +%s
+ now_secs=1587473181
+ date '--date=-30000 min' +%s
+ start_secs=1585673181
+ 0< /tmp/error.log
+ read line
+ printf '%(%#)T' 'Apr 7 05:58:05 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical'
+ 2> /dev/null
+ line_secs=1586253485
+ (( 1586253485 > 1585673181 && line_secs < now_secs || line_secs == now_secs ))
+ egrep -wi '[Cc]ritical'
+ echo Apr 7 05:58:05 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical
+ 1>> /tmp/text
+ read line
+ printf '%(%#)T' 'Apr 7 09:21:12 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical'
+ 2> /dev/null
+ line_secs=1586265672
+ (( 1586265672 > 1585673181 && line_secs < now_secs || line_secs == now_secs ))
+ egrep -wi '[Cc]ritical'
+ echo Apr 7 09:21:12 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical
+ 1>> /tmp/text
+ read line
+ printf '%(%#)T' 'Apr 7 13:05:57 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical'
+ 2> /dev/null
+ line_secs=1586279157
+ (( 1586279157 > 1585673181 && line_secs < now_secs || line_secs == now_secs ))
+ egrep -wi '[Cc]ritical'
+ echo Apr 7 13:05:57 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical
+ 1>> /tmp/text
+ read line

I changed your code in a couple of ways so as to make it easier to debug and see the problem. Putting what you want to do, append to a log file, as a separate statement in an if rather than as part of an && condition, in a trace like the above, you'll see that show up as a separate line. In particular when you see:

+ egrep -wi '[Cc]ritical'
+ echo Apr 7 05:58:05 ip-172-31-19-169 kernel: BIOS-provided physical RAM map: critical

we know that the test succeeded. If you were wondering whether the time test was failing because of the first part (time is not new enough) or the second part (time is too new) or doesn't match the third part (time is now), then putting these in their own if or else would allow a trace to show that.

However I've done something else to make this simpler to debug. Although with a construct like (( $line_secs > $start_secs )) ksh will warn with:

/tmp/bug.ksh: warning: line 14: variable expansion makes arithmetic evaluation less efficient

because it would prefer the more efficient (( line_secs > start_secs )) instead, by adding the extra substitution using $ you'll see the values compared in a trace. And this may help you figure out which part is failing.

If you knew this already and tried that, then I think you'll get more responses if you come to StackOverflow with the trace as shown above, and then the question is narrowed a little bit into "why is this comparision failing?" and you may get more repsonses quicker.

Lastly, I'll mention that I've written debuggers for modern ksh93 newer than 2014-12-24, zsh , and bash . However when I tried the above example on kshdb , it failed and I am not sure why.