Databricks Delta table load take long time to load 1 recod

Question

Whenever databricks notebook is running, I am trying to insert 1 records into a delta table but this is taking around 70 seconds. I am passing start_time as a variable.

val batchDf= Seq((1000, 40, start_time, null, null, status)).toDF("Key", "RunId", "Start_Time", "End_Time", "Duration", "In-progress")

batchDf.write.format("delta").mode("append").saveAsTable("t_audit")

Any idea why loading 1 record into a delta table is taking this long? I would expect this to finish in less than 5 secs.

Answer 1

Databricks is horribly slow in comparison to anything that I have used in the past 30 years, but in your case it could be related to auto optimize