Ways to Trigger a Databricks Notebook

Question

Can someone let me know the possible ways to trigger a Databricks notebook? My preferred method is via Azure Data Factory, but my company is sadly reluctant to deploy ADF at this present moment in time.

Basically, I would like my Databricks notebook to be triggered when a blob is uploaded to Blob store. Is that possible?

Answer 1

You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.

Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly start Auto Loader streams without any permission configurations other than access to your data on cloud storage. In Databricks Runtime 9.1 and above, Auto Loader can automatically detect whether files are arriving with lexical ordering to your cloud storage and significantly reduce the amount of API calls it needs to make to detect new files.

File notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory. File notification mode is more performant and scalable for large input directories or a high volume of files but requires additional cloud permissions for set up.

Refer - https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader