Get partial output from nbconvert.preprocessors.ExecutePreprocessor

Question

Is there a way to get partial output from nbconvert.preprocessors.ExecutePreprocessor? Currently, I use the ExecutePreprocessor to execute my Jupyter notebook programmatically, and it returns the output after executing the entire notebook. However, it would be great to be able to get and save the partial results and while running the notebook. For example, If I have a progress bar in the jupyter notebook, is there a way to continuously read the updated the execution output so that I can see it updating?

This is my current code:

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor

with open('./test2.ipynb') as f:
    nb = nbformat.read(f, as_version=4)
    ep = ExecutePreprocessor(timeout=600, kernel_name='python3')
    ep.preprocess(nb)
    print(nb)
    with open('executed_notebook.ipynb', 'w', encoding='utf-8') as f:
        nbformat.write(nb, f)

however it would be great to be able to continuously read the nb variable and write it to a file while it executes

Answer 1

I ended up doing something like this

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
import threading

f = open('./test2.ipynb')
nb = nbformat.read(f, as_version=4)
ep = ExecutePreprocessor(kernel_name='python3')
def save_notebook():
    threading.Timer(1.0, save_notebook).start()
    with open('executed_notebook.ipynb', 'w', encoding='utf-8') as f:
        nbformat.write(nb, f)
save_notebook()

ep.preprocess(nb)

print('ended')

Seems to work pretty well. If anyone has a better solution feel free to post as well