Create a Job Shared Cluster using Databricks API

Question

I was using a All-Purpose Cluster but after some thought, it was a better option to go with a Job Shared Cluster.The problem is that the AP Cluster was very simple to create via Azure Databricks API, but I can't seem to find any documentation that explains how to do the same with a Job Cluster without the UI ( Job Shared Cluster via UI ).

Using the AWS Documentation, I was able to create this job , but even so, it's not possible to confirm if it's in fact a Job Shared Cluster

So I'd like to know if there's really a way to do this via API and if there is, how?

Answer 1

Any cluster declared in the job_clusters part of the Jobs Create REST API is a shared cluster, and could be used by multiple tasks - they just need to refer to a specific cluster via job_cluster_key setting in the task definition. You can get the job definition via REST API or Databricks CLI and check if that cluster is in the job_clusters section.