Log analytics in Databricks Azure

Question

I have created a log analytics workspace which used the grab all metrics from a databricks cluster.

Actually, my query would be to get the times when the cluster were running more than 10 worker instances. Can I find this information through log analytics in azure for databricks?

any hint would be apprecited.

Answer 1

You can use the Log4j appender to send your Azure Databricks application logs to Azure Log Analytics.

Refer Send application logs using Log4j to know the steps to implement the same.

You can also visualize the Databricks resource metrics using Grafana and Virtual Machine. Check this official Microsoft documentation for the same. Use dashboards to visualize Azure Databricks metrics