使用 Delta 格式覆盖 spark 数据帧写入方法中的特定分区

Question

Able to overwrite specific partition by below setting when using Parquet format, without affecting data in other partition folders

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")

data.toDF().write.mode("overwrite").format("parquet").partitionBy("date", "name").save("abfss://path/to/somewhere")

But this does not work with Delta format in Databricks. Let me know how to handle this in delta format

Answer 1

Create mount between storage account and azure Databricks ,it will create delta storage location. Please follow below syntax.

dbutils.fs.mount(
    source = "wasbs://<container_name>@<Storage_Account_Name>.blob.core.windows.net/",
    mount_point = "/mnt/<Mount_name>",
    extra_configs = {"fs.azure.account.key.<Storage_Account_Name>.blob.core.windows.net":"<Azure_Storage_Access_key>"})
    

then, attach schema df to write option, depending upon schema mention use partitionBy as such . and at the end save data in Mount location where you create delta table .

if you want to read delta formate just change .formate("delta")

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
df2=df.write.option("header",True).partitionBy("DateID","MedallionID").mode("overwrite").format("parquet").save("/mnt/dem123")