Python notebook output format

Question

I'm pretty new to Databricks and I'm playing around with capturing the output from one notebook in another notebook.

Here's my code:

Notebook1

%python
result = dbutils.notebook.run("/01.Mig/SM02. Project /02 Processing Staging/04 User Notebooks/Notebook1", 60)
print("Result: " + result )
if result == 0: dbutils.notebook.exit
else: dbutils.notebook.run("/01.Mig/SM02. Project/02 Processing Staging/04 User Notebooks/Output",60)

Notebook2

%python
resultValue = spark.sql("select count(1) from Notes_Final where record1 like 'GAB%'")
dbutils.notebook.exit(str(resultValue))

The result that gets passed back from Notebook2 is DataFrame[count(1): bigint] . I need it to pass back the value of the count from the SQL in Notebook2 rather than the data type.

What am I missing?

Answer 1

You need to collect the resulting value back to the driver, and even then you might need to print that value out as a string before passing it to dbutils as an exit value.

Your code as is doesnt actually execute a count - it simply creates a DAG to generate the result. Once you call .collect() on your result dataframe it will execute the DAG and give you the count. You can then pass this count as an exit value of the notebook. You may have to pass it as a string by printing it, as I said.