PySpark: How can I suppress %run output in PySpark cell when importing variables from another Notebook?

Question

I am using multiple notebooks in PySpark and import variables across these notebooks using %run path . Every time I run the command, all variables that I displayed in the original notebook are being displayed again in the current notebook (the notebook in which I %run). But I do not want them to be displayed in the current notebook. I only want to be able to work with the imported variables. How do I suppress the output being display every time? Note, I am not sure if it matters, but I am working in DataBricks. Thank you!

Command example:

%run /Users/myemail/Nodebook

Answer 1

This is expected behaviour, when you use %run command allows you to include another notebook within a notebook. This command lets you concatenate various notebooks that represent key ETL steps, Spark analysis steps, or ad-hoc exploration. However, it lacks the ability to build more complex data pipelines.

Notebook workflows are a complement to %run because they let you return values from a notebook. This allows you to easily build complex workflows and pipelines with dependencies. You can properly parameterize runs ( for example, get a list of files in a directory and pass the names to another notebook—something that's not possible with %run ) and also create if/then/else workflows based on return values. Notebook workflows allow you to call other notebooks via relative paths.

You implement notebook workflows with dbutils.notebook methods. These methods, like all of the dbutils APIs, are available only in Scala and Python. However, you can use dbutils.notebook.run to invoke an R notebook.

For more details, refer " Databricks - Notebook workflows ".

Answer 2

You can use the "Hide Result" option in the upper right toggle of the cell: