GCP 云作曲家 PythonOperator 挂起

Question

I am using cloud composer 2.0.23 (airflow 2.2.5) My job is a gen2 cloud function, reading\writing data in BQ. And I send HTTP requests in the operator.

I notice that if I have jobs with execution time more than 10~15min, even if the cloud function finished successfully, the status shown on airflow never updates.

[update1] Cloud function info:

Airflow job status: It should be "success".

Code snippet for calling cloud functions in my DAG file:

def call_cloud_function_endpoint(gcf_url, **kwargs):
    req = urllib.request.Request(gcf_url)
    auth_req = google.auth.transport.requests.Request()
    id_token = google.oauth2.id_token.fetch_id_token(auth_req, gcf_url)
    req.add_header('Authorization', f'Bearer {id_token}')
    response = urllib.request.urlopen(req).read()
    result = json.loads(response)

    return result


PythonOperator(
   task_id='tid', 
   python_callable=call_cloud_function_endpoint, 
   op_kwargs={}
)

Any suggestion is appreciated

Thanks in prior

Answer 1

I have replicated your case with a large-sized public dataset, and it was stuck in "up for retry" or "running" in Airflow. So I changed the Memory allocated and Timeout in Cloud Functions configuration. Try to increase both but I set Timeout into maximum.

It worked for me.

This image is the logs for Airflow.

This is for Cloud Function.

Let me know if this can help or if not, we can still figure out the solution to your problem.