Reading data from Azure Data Lake in C#

Question

I have a requirement to save a large amount (>100GB per day) of transactional data to a data lake gen2. The data is many small JSON transactions so I was planning to batch the transactions together into logical file groups to avoid creating lots of small files. This will allow data analysis to occur over the entire dataset.

However, I also have a separate requirement to retrieve individual transactions from a c# app. Is that possible? There doesn't seem to be an appropriate method on the REST API, and the USQL examples that I've found don't seem to be exposed to c# apps in any way.

Maybe I'm trying to use data lake for the wrong purpose but I don't want to save this quantity of data twice if I can help it.

Thanks!

Answer 1

This solution will allow T-SQL queries against all you JSON files

1. Create a Data Factory to Read JSON files and output parquet formatted files.
2. Use Azure Synapse Workspace On-Demand to read Parque files with OPENROWSET pointing to the Azure Storage location of the parquet files
3. Synapse Workspace On-Demand create a SQL Server Login for C# App
4. Use ADO.NET to send SQL Commands from C#

Answer 2

Synapse Workspace On-Demand create a SQL Server Login for C# App