Databricks notebook hang forever

Question

Actually I am doing some dataframe work for ETL, the dataframe is read from Azure datawarehouse . and seems somehow to notebook hang forever , but I don't know where it is and why it hang so long !!!

Any one has idea and the experience ?

Answer 1

There are various rare scenarios and corner cases that can cause a streaming or batch job to hang. It is also possible for a job to hang because the Databricks internal metastore has become corrupted.

Often restarting the cluster or creating a new one resolves the problem. In other cases, run the following script to unhang the job and collect notebook information, which can be provided to Databricks Support.

For more details, refer " How to Resolve Job Hands and Collect Diagnostic Information ".

Hope this helps.