jupyterhub in docker container not able to connect to external directory

Question

I am setting up a jupyterhub for users in our group to be able to run scripts using our python package on common data without having to install the environment on their computers. The system is based on jupyterhub-deploy-docker ( https://github.com/jupyterhub/jupyterhub-deploy-docker/ ) with the modifications for running on a localhost ( https://github.com/PHI-Toolkit/jupyterhub-deploy-docker-localhost ) which spawns Docker Containers with single user jupyter notebook instances from a dockerized jupyterhub. The idea is to mount a local host directory (or that of a data server) into the jupyterhub container and from there feed it through to the single user instances, to be used as notebook_dir. Then, ipynbs can access the data on the server and be saved there.

The whole approach works well when not dockerized (for testing, authentifying with DummyAuthenticator and spawning with SimpleLocalProcessSpawner), but I can't get the docker containers to see the host directory. My approach is to additionally bind the path on the host machine and specify the used DOCKER_NOTEBOOK_DIR as a target. Therefore, in docker-compose.yml I alter the volumes section

volumes:
  # Bind Docker socket on the host so we can connect to the daemon from
  # within the container
  - "/var/run/docker.sock:/var/run/docker.sock:rw"
  # Bind Docker volume on host for JupyterHub database and cookie secrets
  - "data:${DATA_VOLUME_CONTAINER}"
  - "/path/on/host:${DOCKER_NOTEBOOK_DIR}"

this is used in jupyterhub_config.py as the notebook_dir of the spawner:

notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
# c.DockerSpawner.volumes = {
#     'jupyterhub-user-{username}': notebook_dir,
#     'jupyter-shared': '/home/jovyan/work/shared/',
#     'jupyter-geoserver': '/home/jovyan/work/geoserver',
#     'jupyter-modules': '/home/jovyan/work/modules'
#}
c.DockerSpawner.volumes = { 'jupyter-serverdata': notebook_dir,
    'jupyter-shared': '/home/jovyan/work/shared/',
}

I would think that then the ipython notebook should access the contents of /path/on/host, but I still get the same directories as in the standard configuration (with the same content, even though I removed the jupyterhub containers and images). As I suspect this is due to the DockerSpawner, I tried to use the SimpleLocalProcessSpawner, but still running jupyterhub in its docker container. However I couldn't get this to run (spawning results in error 500).

Any help on how to feed an external path (host path) through a jupyterhub running in a docker container into a jupyter notebook instance would be greatly appreciated.

Answer 1

Looking into the documentation and code often helps. I got it to work:

In the dockerspawner source code https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py#L218 the option of mounting host file/directory into container (however the description was not detailed enough for me to understand how. To that end, the tests were helpful: https://github.com/jupyterhub/dockerspawner/blob/3906f4bebc92b383c73fb8d06c58a7c57003939a/tests/volumes_test.py#L25 suggests that specifying

notebook_mount_dir = '/path/on/host'
notebook_dir = '/path/in/dockerinstance'
c.DockerSpawner.volumes = {notebook_mount_dir: {"bind": notebook_dir, "mode": "rw"}}

should make it work and indeed it does. Hope this helps anyone in the future.