How can we sort threads on Linux by their System CPU (not the overall or User CPU)?

Question

Top can be used to sort processes/threads by overall CPU. How about sorting threads just by their system CPU? I am trying to investigate a case of high system CPU on a server, and strace gives me the breakup of system time in terms of system calls made by a thread. But, how does one identify the thread that contributes the most to System CPU (the thread that makes the most expensive syscalls, essentially)?

Answer 1

syscalls as reported by strace may or may not have correlation with the time spent actively doing something in the kernel. In fact you don't want to look at this most of the time.

What you want instead is a flamegraph and go from there. In particular, it may be kernel is spending a lot of time not even executing any syscalls in the first place.

http://www.brendangregg.com/flamegraphs.html