如何更新 Azure 中放置在 Blob 存储中的 CSV 文件

Question

I have Blob Storage in Azure that holds a CSV file. While incrementally updating it creates a new CSV file instead we need to update the existing CSV file.

Is there any solution in the Azure data factory to update incremental data into the existing CSV file?

Answer 1

You can work on incremental updates to a Blob Storage file using Azure Databricks and in ADF you can call the databricks notebook activity

In this you will have to mount the Azure storage on databricks cluster first

Then you will have to read the original and incremental file or data as a dataframe

You can perform joins on the files and create an incremental data frame which you can overwrite to your original file which will be your incremental data and that too in a single file