如何处理最早到最新的bash文件？

Question

Overview

I have a bunch of log files which rollover when they reach a certain size. Each line in the log file has a bunch of logger formatting and then some interesting information. I want to take those files and remove the formatting from the beginning of each line and then put the output of all of that into a single file. I will then eventually take that one file and load it into another application manually.

Details

The file structure looks something like this:

logs
 |-- modules
 |    +-- ...
 |-- application.log
 |-- gc.log
 |-- gc.log.1
 |-- ...
 +-- gc.log.10

So logs contains subdirectories and multiple log files. The ones I am insterested are gc.log* .

Each gc.log* file rolls over to a new file when it gets full. gc.log is always the newest and it goes up to gc.log.10 being the oldest (by default there are only 10, max version 9, but this is configurable).

A typical gc.log* contains thousands of entries like:

INFO   | jvm 1    | 2015/05/28 04:40:58 | 1164752.977: [GC pause (young), 0.06583700 secs]
INFO   | jvm 1    | 2015/05/28 04:40:58 |    [Parallel Time:  45.2 ms]
INFO   | jvm 1    | 2015/05/28 04:40:58 |       [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
INFO   | jvm 1    | 2015/05/28 04:40:58 |        Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]
...

(Yes these are G1 GC logs from the Oracle JVM. It is these that I need in a separate file so I can graph with GCViewer)

Once I have stripped out the formatting I need it to look like:

1164752.977: [GC pause (young), 0.06583700 secs]
   [Parallel Time:  45.2 ms]
      [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
       Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]

What I have so far

So far I have learnt that I shouldn't be using ls to get the files. I found this on another SO question (sorry I forgot which one) Why you shouldn't parse the output of ls(1) .

I am using the following to list the files and then sort them from oldest to newest:

find "$logDir" -maxdepth 1 -type f -name 'gc.log*' | sort -Vr

Which gives me the following:

./gc.log.10
./gc.log.9
./gc.log.8
./gc.log.7
./gc.log.6
./gc.log.5
./gc.log.4
./gc.log.3
./gc.log.2
./gc.log.1
./gc.log

The command I have to remove the formatting is:

sed -e 's/^.\{7\}[|].\{10\}[|].\{21\}[|] //g'

(I may just use cut -c43- )

Problem

I'm not sure how to get the output from sort into sed .

The following doesn't work when the file name (or $logDir ) has spaces:

find "$logDir" -maxdepth 1 -type f -name 'gc.log*' | sort -Vr | xargs sed -e "s/^.\{7\}[|].\{10\}[|].\{21\}[|] //g"

I'm also going to need to take the output from sed and then concatenate that all together into a single file.

Question

Finally the question:

• How can I list certain files in a directory in reverse natural number sort order, remove a pattern from the beginning of each line in those files and lastly concatenate the results into a single file (in bash)?

Answer 1

Since your filenames are fixed, you can simply use brace expansion:

for wrapper in wrapper.log{.{9..1},}; do
    echo "$wrapper"
    # do whatever you want to do...
done

For your purpose, I guess, this could work too:

$ cat wrapper.log{.{9..1},} | sed ...

A bit more generic version:

$ logfile="wrapper.log" # may contain spaces in filename
$ cat "$logfile"{.{9..1},} | sed ...

Answer 2

In this case your file names are so simple and you're doing so little with them, I'd be tempted to just use the ls output, assuming your files have the intuitive progressive modification times then all you'd need is:

ls -rt gc.? gc | xargs awk -F' [|] ' '{print $NF}' > newfile

For example:

$ cat gc
INFO   | jvm 1    | 2015/05/28 04:40:58 | 1164752.977: [GC pause (young), 0.06583700 secs]
INFO   | jvm 1    | 2015/05/28 04:40:58 |    [Parallel Time:  45.2 ms]
INFO   | jvm 1    | 2015/05/28 04:40:58 |       [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
INFO   | jvm 1    | 2015/05/28 04:40:58 |        Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]
$
$ cat gc.1
INFO   | jvm 1    | 2015/05/28 04:40:58 | 1234567.977: [GC pause (young), 0.06583700 secs]
INFO   | jvm 1    | 2015/05/28 04:40:58 |    [Parallel Time:  45.2 ms]
INFO   | jvm 1    | 2015/05/28 04:40:58 |       [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
INFO   | jvm 1    | 2015/05/28 04:40:58 |        Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]
$
$ cat gc.2
INFO   | jvm 1    | 2015/05/28 04:40:58 | 8889996.977: [GC pause (young), 0.06583700 secs]
INFO   | jvm 1    | 2015/05/28 04:40:58 |    [Parallel Time:  45.2 ms]
INFO   | jvm 1    | 2015/05/28 04:40:58 |       [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
INFO   | jvm 1    | 2015/05/28 04:40:58 |        Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]

$ ls -rt gc.? gc | xargs awk -F' [|] ' '{print $NF}'
8889996.977: [GC pause (young), 0.06583700 secs]
   [Parallel Time:  45.2 ms]
      [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
       Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]
1234567.977: [GC pause (young), 0.06583700 secs]
   [Parallel Time:  45.2 ms]
      [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
       Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]
1164752.977: [GC pause (young), 0.06583700 secs]
   [Parallel Time:  45.2 ms]
      [GC Worker Start (ms):  1164752977.7  1164752977.7  1164752977.7  1164752977.9
       Avg: 1164752977.8, Min: 1164752977.7, Max: 1164752977.9, Diff:   0.2]

Answer 3

If you want to do it right (and have GNU find and sort ), tell find to write the filenames preceded by mtime and separated by NUL characters (the only character which can't exist in a file's fully qualified path on UNIX); use sort to sort by mtime (rather than trying to mess with names); and then read both pieces of data in:

while IFS= read -r -d ' ' mtime && IFS= read -r -d '' filename; do
  sed -e 's/^.\{7\}[|].\{10\}[|].\{21\}[|] //g' <"$filename"
done < <(find "$logDir" -maxdepth 1 -type f -printf '%T@ %P\0' | sort -nz)

This will process files in order from oldest to newest.