What is the easiest way to pull data from a blob and load it into a table in SQL Server?

Question

I have hundreds of zipped files sitting in different folders, which I can access using MS Storage Explorer. I just setup a SQL Server DB in Azure. Now I am trying to figure out how I can pull data from each file in each folder, unzip it, parse it, and load it into tables. The data is coming in daily, so the folders are named '1', '2', '3', etc. '31', for the days of the month. Also, I have monthly folders '1' through '12', for the 12 months of the year. Finally, I have folders named '2017', '2018', and '2019'. I could ask some people on my team to write Python code to do the work, but it seems like there should be an easier way. I would like to avoid writing thousands of lines of Python code, if there is a simpler way to do this kind of thing. TIA.

Answer 1

I would create Azure Logic App that would

1. HTTP trigger (this will be manually ran)
2. Iterate through all blobs present on storage account
3. For each element use blob connector which has action called Extract archive to extract zipped files into blob
4. Use Azure SQL BCP to pull this, if this is different format than CSV then use Azure Data Factory

Later if you expect new blobs do similar flow except use New or modified blob trigger.

For Azure SQL BCP from BLOB example with Logic Apps check this article https://marczak.io/posts/azure-loading-csv-to-sql/

And if you want general intro into Azure Logic Apps check this video https://youtu.be/ZvsOzji_8ow In here there is also new blob trigger demo.

In case you need data factory feel free to check this video. It has blob to sql demo too. Also you can trigger data factory from logic apps too. https://youtu.be/EpDkxTHAhOs