如何将脚本路径作为数据块笔记本中的变量传递给 %run magic 命令？

Question

I want to run a notebook in databricks from another notebook using %run . Also I want to be able to send the path of the notebook that I'm running to the main notebook as a parameter.
The reason for not using dbutils.notebook.run is that I'm storing nested dictionaries in the notebook that's called and I wanna use them in the main notebook.

I'm looking for Something like:

path = "/References/parameterDefinition/schemaRepository"

%run <path variable>

Answer 1

You can pass arguments as documented on Databricks web site: https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run

In the top notebook you can call

%run /path/to/notebook $X="10" $Y="1"

And then in the sub notebook, you can reference those arguments using the widgets API as in

x_value = dbutils.widgets.get("X")
y_value = dbutils.widgets.get("Y")

To your specific question, it would look something like this where "path" is the variable to be referenced via the widgets API in the target notebook:

 %run /path/to/notebook $path="/path/to/notebook"

Answer 2

Magic commands such as %run and %fs do not allow variables to be passed in.

The workaround is you can use dbutils as like dbutils.notebook.run(notebook, 300 ,{})

Answer 3

Unfortunately it's impossible to pass the path in %run as variable. You can pass variable as parameter only, and it's possible only in combination with with widgets - you can see the example in this answer . In this case you can have all your definitions in one notebook, and depending on the passed variable you can redefine the dictionary.

There will be a new functionality coming in the next months (approximately, see public roadmap webinar for more details) that will allow to import notebooks as libraries using the import statement. Potentially you can emulate the same functionality by exporting the notebook into the file on disk using the Export command of Workspace API , decoding the data & importing file's content, for example, if you have notebook called module1 with content

my_cool_dict = {"key1": "abc", "key2": 123}

then you can import it as following:

import requests
import base64
import os


api_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()
host_token = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()

path = "/Users/..../module1"

# fetch notebook
response = requests.get(f"{api_url}/api/2.0/workspace/export",
    json = {"format": "SOURCE", "path": path},
    headers={"Authorization": f"Bearer {host_token}"}
  ).json()

# decode base64 encoded content
data = base64.b64decode(response["content"].encode("ascii"))

# write the file & __init__.py, so directory will considered a module
dir = os.path.join("/tmp","my_modules")
if not os.path.exists(dir):
    os.mkdir(dir)

with open(os.path.join(dir, os.path.split(path)[-1]+".py"), "wb") as f:
  f.write(data)
with open(os.path.join(dir, "__init__.py"), "wb") as f:
  f.write("\n".encode("ascii"))

# add our directory into system path
import sys
sys.path.append(dir)

# import notebook
from module1 import my_cool_dict

and see that we got our variable:

Answer 4

Did you try %run $path ? I believe it should work.

Answer 5

Problem

You can't pass it as a variable while running the notebook like this:

In notebook1:

path_var = "/some/path"

%run ./notebook2
%path=path_var

Solution

However what you can do, and what I did, is access the dbutils object or the variable of notebook1 in notebook2:

In notebook1:

dbutils.widgets.text("path","", "")
path_var = "/some/path"

%run ./notebook2

Then in notebook2:

"""
No need to define path widget in Notebook2 like this:
dbutils.widgets.text("path","", "")
"""
path = dbutils.widgets.get("path")
print(path)

Output: /some/path

"""
Or you can access the path_var of Notebook1 directly
without defining it anywhere in Notebook2 like this 
"""
print(path_var)

Output: /some/path

So this helps when you are using complicated variables such as heavily nested dictionaries8 in the notebook.

Benefit

What I love about this approach is that environment of notebooks get shared when you call a notebook, meaning you can access variables & methods of Notebook1 in some Notebookn and vice versa such that:

• Notebook1 is calling Notebook2
• Notebook2 is calling Notebook3.....
• Notebookn-1 is calling Notebookn