How to Handle or Architecture, incremental data ingestion in Azure data lake Store?

Question

I've Two Custom code dll, for Image related to IP Cams.

dll-One : Extract image from IP cams and can be stored it to Azure data lake Store.

Like :

• /adls/clinic1/patientimages
• /adls/clinic2/patientimages

dll-two : Use those image and extract information from it and load data into RDBMS tables.

So for instance in RDBMS ,say there are entities dimpatient, dimclinic and factpatientVisit.

For start, a one time data can be exported to defined location in Azure data lake store.

Like:

• /adls/dimpatient
• /adls/dimclinic
• /adls/factpatientVisit

Question : How to push incremental data in same file or how we can handle this incremental load in Azure data Analytics?

This like implementing Warehouse in Azure Data Analytics.

Note : Azure SQL db or any other storage offered by Azure is not want to. I mean why to spend in other Azure Services if one type of storage has capabilities to hold all types of data.

adls is name of my ADLS storage.

Answer 1

I am not sure I completely understand your question, but you can organize your data files in Azure Data Lake Store or your rows in partitioned U-SQL tables along a time dimension, so you can add new partitions/files for each increment. In general, we recommend that such increments are of substantial sizes though to preserve the ability to scale.