在 Airflow 1.10 重试创建 dataproc 集群

Question

Hello all I need some help is Airflow.We use Airflow 1.10.We have a requirement where we want to retry the task of creating and deleting the dataproc cluster if the task fails.We do have retry parameter in Airflow 2.0 but we do not have any such parameter which retry's creating and deleting the cluster in airflow 1.10.Can anyone suggest any alternative so that if creation or deletion of dataproc cluster fails,then we can retry creating it.

Answer 1

DataprocCreateClusterOperator has retry parameter:

 :param retry: A retry object used to retry requests. If ``None`` is specified, requests will not be retried.

This ability was added in PR and available for Airflow>=1.10

If you are using Airflow<2.0: You will need to:

pip install apache-airflow-backport-providers-google

If you are using Airflow>=2.0: You will need to:

pip install apache-airflow-providers-google

Then you can import the operator as:

from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator

and use it as:

create_cluster_operator = DataprocCreateClusterOperator(
    task_id='create_dataproc_cluster',
    cluster_name="test",
    ...,
    retry=YOUR_RETRY_VALUE
    timeout=YOUR_TIMEOUT_VALUE
)

Note that all Airflow operators inherits from BaseOperator which has retries parameter:

 :param retries: the number of retries that should be performed before failing the task

Don't be confused between retry and retries .