Upload to Azure Storage container from FileServer using Azure databricks

Question

I want to upload binary files from Windows FileSystem to Azure blob. I achieved it with Azure data factory with the below steps

1. Installed integration run time on the FileSystem
2. Created a linked service in ADF as FileSystem
3. Created a binary dataset with the above linked service
4. Use CopyData activity in a ADF Pipeline, set the binary dataset as source and Azure Blob as Sink

Post upload, I am performing some ETL activities. So my ADF pipeline has two components,

1. Copy Data
2. Databricks Notebook

I am wondering if I could move the Copy Data fragment to Databricks? Can we upload binary files from Windows FileSystem to Azure blob using Azure Databricks?

Answer 1

I think it is possible, you may have to do network changes https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/on-prem-network