Responsive IPython notebook with running progress bar of dask/distributed

Question

I am running a cluster with dask.distributed . Currently I submit tasks to the cluster with Jupyter notebook that I use as a GUI.

The respective notebook cell contains the following code.

%pylab inline
%load_ext autoreload
%autoreload 2

from distributed import progress
sys.path.append('/path/to/my/python/modules/on/NAS')

import jobs
jobid = jobs.add_new_job(...)
r = jobs.start_job(jobid)
progress(r)   

jobs is the name of my python module. jobs.add_new_job returns a string with job identifier. jobs.start_job returns a list of distributed.client.Future s. Final result of the job is a report with some numbers and some plots in PDF.

Now I'd like to implement a job queue with some indication on what is being processed now and what is waiting.

My goal is to implement the following scenario.

Member of my team prepares some data for a new job, then opens Jupyter notebook in his browser, enters job parameters in the cell in the call to add_new_job , then executes this cell, then closes that page and waits till computations are done. He could also leave the page open and observe the progress.

Up to now I've found that if I submit a single job to a cluster by running the cell once and waiting till everything is done, everything works like a charm.

If I try to submit one more job by simply editing the cell code and running it again, then the cluster stops calculating first submitted job. My explanation of this is that r is deleted and its destructor sends cancellation requests to the cluster.

If I try to submit a new job by making a copy of the notebook, a new empty page opens in the browser, and then it takes very long time until the notebook loads and allows a user to do anything.

Also, progress bar, displayed by progress , very often disappears by itself.

I've read already about JupyterHub, but currently it seems to me that using it is like a shooting sparrows with a heavy artillery, and there should be more simple way.

Answer 1

My explanation of this is that r is deleted and its destructor sends cancellation requests to the cluster

This is correct. A simple way to avoid this would be to add r to some result set that is not deleted every time you run your cell

-- cell 1 -- 

results = []

-- cell 2 --

import jobs
jobid = jobs.add_new_job(...)
r = jobs.start_job(jobid)
results.append(r)
progress(r)