我在 blob 存储中获得连续的 blob 文件。 我必须加载 Databricks 并放入 Azure SQL DB。 用于编排此管道的数据工厂

Question

I receive data continuously in blob storage. I have initially 5 blob files in the blob storage I'm able to load from blob to Azure SQL DB using Databricks and automated it using Data factory, But the problem is when newer files come in blob storage the databricks loads these files along with older files and sends it into Azure SQL DB. I don't want these old files, every time I want only the newer one's, so that same data is not loaded again and again in the Azure SQL DB.

Answer 1

Easiest way to do that is to simply archive the file that you just read into a new folder name it archiveFolder . Say, your databricks is reading from the following directory:

mnt
  sourceFolder
    file1.txt
    file2.txt
    file3.txt

You run your code, you ingested the files and loaded them in SQL server. Then what you can simply do is to archive these files (move them from the sourceFolder into archiveFolder . This can simply be done in databricks using the following command

dbutils.fs.mv(sourcefilePath, archiveFilePath, True)

So, next time your code runs, you will only have the new files in your sourceFolder .