How to trigger a pipeline in Azure Data Factory v2 or a Azure Databricks Notebook by a new file in Azure Data Lake Store gen1

Question

I am using a Azure Data Lake Store gen1 for storing JSON files. Based on these files i have Notebooks in Azure Databricks for processing them. Now i want to trigger such a Azure Databricks Notebook when a new file is creating in Azure Data Lake Store gen1 . I couldnt find any Trigger which could do this. do you know any way?

Answer 1

Currently, this is not yet implemented/Supported by Microsoft . But it is on their Roadmap(I believe).

You can do this in 2 ways,

1. Azure Functions(through Event Grid)

2. Logic Apps

---

Option #1

Currently, Microsoft is building on #1.

You can track the issue here .

As per this

This feature is not a high priority for us right now, but I will note that the announcement for Azure Event Grid listed Data Lake as one of the integrations they are building. Once you can subscribe to Data Lake updates through Event Grid, running an Azure Function would be trivial (see here for some info).

You can vote your voice to support the event grid (provider) in DataLake.

---

Option #2

This is also not yet implemented, but you can Upvote your voice here to support this feature