Databricks is "Updating the Delta table's state"

Question

I'm reading and joining multiple delta tables from a Datalake and store the result back to another Deltalake location. When doing so, Databricks is showing me :

Depending on how many delta tables I join with each other, this can take up to very long time. Even tough the joining itself would just take up to a few minutes, the state update takes up to an hour.

What is happening when I see Updating the Delta table's state ? Can I somehow optimize this?

Answer 1

Thank you Karthikeyan Rasipalay Durairaj , Posting your suggestion as an answer to help other community members.

Updating the Delta table's state.

The command status report means ,

• At the beginning of each query Delta tables auto-update to the latest version of the table.
• Delta Lake writes checkpoints as an aggregate state of a Delta table at an optimized frequency.
• Databricks optimizes the performance of higher-order functions and DataFrame operations using nested types.
• For Delta Lake on Databricks SQL optimization command reference information, refer
  • Databricks Runtime 7.x and above: OPTIMIZE (Delta Lake on Databricks)
  • Databricks Runtime 5.5 LTS and 6.x: Optimize (Delta Lake on Databricks)