Python profile: How to get the time for each individual execution (no aggregation)

Question

I am trying to debug a python based project that involves many repeated calls to the same set of functions. This code is part of an API.

However, over time the execution time for these functions increases linearly. I now need to find out what is causing this increase of total time a single request is taking to complete. Is there a way to profile python code and get the output for each execution of repeatedly called functions?

As an abstract example: Let's say function 1 involves calls to function 1, function 2, function 3. I call function 1 10,000 times and then notice that the execution of function 1 took initially 0.5 seconds but increases to 5 seconds at the 10,000 call to it. How can I figure out whether function 1, function 2 or function 3 is causing this increase?

Answer 1

You can use Linux perf tools if it is on Linux environment and check number of cycles for each function, number of instructions, L1 cache load, L1 cache miss and L2 cache load/miss and compare them one by one.

On the below example you can find the examples comparing different functions.