Load all CSVs from path on local drive into AzureSQL DB w/Auto Create Tables

Question

I frequently need to validate CSVs submitted from clients to make sure that the headers and values in the file meet our specifications. Typically I do this by using the Import/Export Wizard and have the wizard create the table based on the CSV (file name becomes table name, and the headers become the column names). Then we run a set of stored procedures that checks the information_schema for said table(s) and matches that up with our specs, etc.

Most of the time, this involves loading multiple files at a time for a client, which becomes very time consuming and laborious very quickly when using the import/export wizard. I tried using an xp_cmshell sql script to load everything from a path at once to have the same result, but xp_cmshell is not supported by AzureSQL DB.

https://learn.microsoft.com/en-us/azure/azure-sql/load-from-csv-with-bcp

The above says that one can load using bcp, but it also requires the table to exist before the import... I need the table structure to mimic the CSV. Any ideas here?

Thanks

Answer 1

If you want to load the data into your target SQL db, then you can use Azure Data Factory[ ADF ] to upload your CSV files to Azure Blob Storage, and then use Copy Data Activity to load that data in CSV files into Azure SQL db tables - without creating those tables upfront .

ADF supports 'auto create' of sink tables. See this , and this