How can I reference the path of a notebook in Databricks/what is %run doing?

Question

I'm familiar with the %run magic commands in Databricks, but where do the notebooks actually live?

The rootdir using %sh pwd appears to be /databricks/driver. Making Python look for the notebook path (eg, subprocess.call([the/notebook/path]) fails, since it looks for the notebook path in this rootdir and obviously comes up empty. The notebooks must live somewhere else, but where? What path is %run invoking to find the notebook?

For reference, I am trying to implement pytest within Databricks, and pytest.main() should allow me to run the tests within the notebook itself in lieu of the typical command line approach... if it could just find out where the notebook lives.

Answer 1

I was looking for this same thing and after some digging into dbutils , this is the solution I came up with:

import json
context = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())
print(context['extraContext']['aclPathOfAclRoot']) # Base path
print(context['extraContext']['notebook_path']) # Notebook path

Answer 2

You can get the current Notebook Path according to https://docs.azuredatabricks.net/user-guide/faq/get-notebook-path.html by using Scala. For Python you need the workaround via Scala and a Widget as described in the same FAQ:

%scala
dbutils.widgets.text("notebook", dbutils.notebook.getContext().notebookPath.get)

And read it in python:

%python
dbutils.widgets.get("notebook")

Answer 3

Based on @Martin's answer, but without using a widget (which creates a UI element that you might not want):

%scala
spark.conf.set("nb.path", dbutils.notebook.getContext().notebookPath.get)

And then:

%python
path = spark.conf.get("nb.path")