Tool to work with log file produced by parallel workers (e.g. by build system)

Question

We have a build system that does write logs from parallel workers to the same file. I'm looking for the tool to simplify life with this type of logs.

Each line in this log file is arbitrary text sting, starting with worker id. Eg:

11> compiling my.cpp
14> compiling your.cpp
7> ***starting linking***
7> Linking project CoreEngine...
14> compiling my2.cpp

• 11, 14, and 7 do unrelated job here.

Sometimes log records produced by a single worker are thousands lines apart. I'm looking for a way to optimize this experience. Would be nice to hide all unrelated records, and just to see the interesting one.

As a bonus - sometimes worker can start a new worker with new ID with some constant text message. Would be nice to track these dependencies somehow too.

Is anyone aware of a tool that could help with this?

I did search in the Internet and here, not obvious answer yet.

thank you!

Answer 1

If you're looking for a tool, grep, is probably the simple one here. You can extract out individual worker threads.

If you can code in just about any language, you would be better off writing your own parser and output the data the way you want in a format that is most readable.