Inter-thread communication with python: Plotting memory-consumption using separate python thread

Question

Withing a python-script I call sequentially different functions (lets say func_a, func_b, func_c), which process a given set of input data.

The execution takes about 30min.

Within these 30 minutes, I want to track and plot the memory consumption of my program.

I considered to do this within a separate thread (eg my_tracking_thread), which checks for the current memory usage.

Eg like this:

import psutil
process = psutil.Process(os.getpid())
mem = process.get_memory_info()[0] / float(2 ** 20)

Plotting the collected data of my_tracking_thread with matplotlib, I would like to include as additional information the time-stamps, at which the different functions started (func_a@4:14, func_b@6:23, func_c@25:48).

Therefore the question: How do I get my_tracking_thread to notify, that func_{a|b|c} has been started?

Any help appreciated.

Answer 1

It's difficult to see how you could do that from the other thread. Do you need to? I'd suggest wrapping them in decorators that log the time they start and finish, and then when you generate the plot with the memory consumption data merging in the start/finish information.

import collections
import functools
import time
LogEvent = collections.namedtuple('LogEvent', 'function event timestamp')

events = []

def log_start_finish(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        events.append(LogEvent(func.__name__, 'start', time.time())
        result = func(*args, **kwargs)
        events.append(LogEvent(func.__name__, 'finish', time.time())
        return result
    return wrapper

@log_start_finish
def func_a(...):
    ....

@log_start_finish
def func_b(...):
    ....

@log_start_finish
def func_c(...):
    ....