[英]Create Azure databricks notebook from storage account
We have python script stored in Azure storage account in blob.我们将 python 脚本存储在 Blob 中的 Azure 存储帐户中。 We want to deploy / create this python script (as notebook) in azure databricks cluster so later we can run Azure data factory pipeline and pipeline can execute notebook created/deployed in databricks.我们想在 azure databricks 集群中部署/创建这个 python 脚本(作为笔记本),以便稍后我们可以运行 Azure 数据工厂管道和管道可以执行数据块创建/部署的笔记本。
We want to create / deploy this script only one time as and when its available in blob.我们只想在 blob 中可用时创建/部署此脚本一次。
I have tried to search over the web but couldn't find proper solution for this.我试图搜索 web 但找不到合适的解决方案。
Is it possible to deploy/create notebook from storage account?是否可以从存储帐户部署/创建笔记本? if yes, how?如果是,如何?
Thank you.谢谢你。
You can import notebook into Databricks using the URL, but I expect that you won't make that notebook public.您可以使用 URL 将笔记本导入 Databricks,但我希望您不会公开该笔记本。
Another solution would be to use a combination of azcopy
tool with Databricks CLI ( workspace
sub-command).另一种解决方案是结合使用azcopy
工具和Databricks CLI ( workspace
子命令)。 Something like this:像这样的东西:
azcopy cp "https://[account].blob.core.windows.net/[container]/[path/to/script.py" .
databricks workspace import -l PYTHON script.py '<location_on_databricks>'
You can also do it completely in notebook, combining the dbutils.fs.cp
command with Databricks's Workspace REST API , but that's could be more complicated as you need to get personal access token, base64 the notebook, etc.您也可以完全在笔记本中完成此操作,将dbutils.fs.cp
命令与 Databricks 的工作区 REST API相结合,但这可能会更复杂,因为您需要获取个人访问令牌 Z95A14046A71A207E4AFCD5878
We can use databricks API 2.0 to import python script in databricks cluster.我们可以使用databricks API 2.0 在databricks集群中导入python脚本。
Here is the API definition: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/workspace#--import这是 API 定义: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/workspace#--import
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.