linux perf report inconsistent behavior

Question

I have an application I'm profiling using perf and I find the results when using perf report are not consistent, and I can't discern the pattern.

I start the application and profile it by pid for 60 seconds:

perf record -p <pid> -o <file> sleep 60

And when I pull the results in with perf report -i <file> , sometimes I see a "+" in the far left column that allows me to drill down into the function call trees when I press ENTER, and sometimes that "+" is not there. It seems to be dependent on some property of the recorded file, in that I have a collection of recorded files, some which allow this drill down and some which do not.

Any suggestions on how to get consistent behavior here would be appreciated.

Answer 1

The default event being measured by perf record is cpu-cycles. (Or depending on the machine, sometimes cpu-cycles:p or cpu-cycles:pp)

Are you sure your application is not sleeping a lot? Does it consume a lot of cpu cycles?

Try a perf measurement on something that stresses the CPU by doing a lot of computations:

$ apt-get install stress
$ perf record -e cpu-cycles --call-graph fp stress --cpu 1 --timeout 5
$ perf report

Subsequent runs should then show more or less similar results.

In case your program is CPU intensive, and call stacks do differ a lot between runs, then you may want to look at the --call-graph option, as perf can record call-graphs with different methods:

• fp (function pointer)
• lbr (last branch record)
• dwarf

Maybe different methods give better results.