如何在 Azure 数据块中自动化长时间运行的代码和保存数据？

Question

I am using the %run feature in Azure databricks to execute many notebooks in sequence from a command notebook. One notebook has a long computation on a dataset (~ 5 hrs) and I want to save the output of this. I tried including the save step at the end of the long-running notebook, but the save times out (see error below). I'm only seeing this error when the long-running notebook takes 2+ hrs to run. Is there any way I can automate this?

I'm able to pass the data I want back through the %run feature in the command notebook and save the data there, but I have to run the save manually after the long-running notebook, otherwise I get the same authentication timeout error. I'd like to be able to have one notebook where I only need to click "run all".

Answer 1

I find it is better to break up long notebooks into smaller ones and use the multi-task job scheduler to help run them in order.