Create Azure databricks notebook from storage account

Question

We have python script stored in Azure storage account in blob. We want to deploy / create this python script (as notebook) in azure databricks cluster so later we can run Azure data factory pipeline and pipeline can execute notebook created/deployed in databricks.

We want to create / deploy this script only one time as and when its available in blob.

I have tried to search over the web but couldn't find proper solution for this.

Is it possible to deploy/create notebook from storage account? if yes, how?

Thank you.

Answer 1

You can import notebook into Databricks using the URL, but I expect that you won't make that notebook public.

Another solution would be to use a combination of azcopy tool with Databricks CLI ( workspace sub-command). Something like this:

azcopy cp  "https://[account].blob.core.windows.net/[container]/[path/to/script.py" .
databricks workspace import -l PYTHON script.py '<location_on_databricks>'

You can also do it completely in notebook, combining the dbutils.fs.cp command with Databricks's Workspace REST API , but that's could be more complicated as you need to get personal access token, base64 the notebook, etc.

Answer 2

We can use databricks API 2.0 to import python script in databricks cluster.

Here is the API definition: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/workspace#--import