Linux shell command to split a log file

Question

I was wondering if there is a better way to split a log text file than doing the shell loop below, ideally with a single shell command.

The log file looks like that:

2016-11-20T16:19:21+00:00 Logging started
2016-11-20T16:20:41+00:00 System is up
2016-11-20T16:21:07+00:00 Unknown event 45
...
2016-11-25T08:40:00+00:00 Blah blah
2016-11-25T08:42:00+00:00 Blah blah
...
2016-11-27T11:32:00+00:00 System powering down

• All the lines start with an ISO8601 date marker (UTC)
• The lines are in chronological order, by construction, because lines are appended when there is an event to log.
• The file is growing

So, the task we want to accomplish is split the file at a given time. Let's say I keep only the last week entries to avoid the ever-growing syndrome.

So having a date of '2016-11-25T08:41:00+00:00', I wish to keep only those entries posterior to that date. Note that the date at which we want to cut our file does not necessary correspond to an existing entry (as in the example).

So, the best I could do is a piece of code like that:

WHEN='2016-11-25T08:41:00+00:00' # actually that is read as a parameter

while read line; do
  if [ "${line}" \> "${WHEN}" ]; then
    echo "${line}"
  fi
done <"${LOGFILE}" >"${CUTFILE}"

That works, but as it is a shell loop, it could be slow if the file becomes really big.

So, any better suggestion with a standard command/utility?

Answer 1

You can use the following command:

tail -n +$(cat logfile | grep -m1 -n "2016-11-25" | cut -d: -f1) logfile 

-m1 option of grep will match the first occurence of pattern (2016-11-25) and -n will print the corresponding line number along with the matching pattern.

eg logfile from above

root@ubuntu:/home# cat logfile 
2016-11-20T16:19:21+00:00 Logging started
2016-11-20T16:20:41+00:00 System is up
2016-11-20T16:21:07+00:00 Unknown event 45
2016-11-25T08:40:00+00:00 Blah blah
2016-11-25T08:39:02+00:00 Blah blah
2016-11-25T08:39:04+00:00 Blah blah
2016-11-25T08:42:00+00:00 Blah blah
2016-11-27T11:32:00+00:00 System powering down

grepping the required pattern "2016-11-25"

root@ubuntu:/home# cat logfile | grep -m1 -n "2016-11-25"
4:2016-11-25T08:40:00+00:00 Blah blah

to take the first value of the above output use cut command with delimiter":"

root@ubuntu:/home# cat logfile | grep -m1 -n "2016-11-25" | cut -d: -f1
4

and pass this to tail command, "tail -n +" to get your required output

root@ubuntu:/home# tail -n +$(cat logfile | grep -m1 -n "2016-11-25" | cut -d: -f1) logfile
2016-11-25T08:40:00+00:00 Blah blah
2016-11-25T08:39:02+00:00 Blah blah
2016-11-25T08:39:04+00:00 Blah blah
2016-11-25T08:42:00+00:00 Blah blah
2016-11-27T11:32:00+00:00 System powering down