使用 xcom_pull 拉取相同任务推送的键值 - Airflow

Question

I've built an Airflow Operator that executes HTTP request against a cloud API. After executing a request I get a run_id that identifies the execution, and then keep checking my request status until it finish (it can take some time).

I'm trying to develop a mechanism that will save this run_id to xCom, for cases of network issues/ spot taken, when my pod loose connection with the cloud service. I'm using Airflow retries, so I want to resume the same connection after the failure and keep checking the status of the run_id that saved to xCom.

I was able to push the run_id to xCom, but I when I pull it for the same task I get None instead of the value that I pushed and see in the Admin dashboard (I could pull it for other tasks in the pipeline).

• Is there any limitation for getting xCom pushed value that was pushed from the same task (in earlier try) that I'm trying to pull from?
• I have also an option to use airflow Variables instead of xCom, but it looks like it's not relevant for my use-case but for sharing data between dags (and in my use case I want to share data between task to itself in future run).

I execute this code inside the execute function of my operator: This is how I pushed the value:

self.xcom_push(context, key='run_id', value='111')

This is how I pulled the values:

value = self.xcom_pull(context, key='run_id', task_ids=self.task_id, include_prior_dates=True)

but value is always None (unless I try to pull data of other task (but it's not relevant for my use case).

Answer 1

I used airflow Variables instead of xCom, since it's not possible to get xCom pushed value that was pushed from the same task (in an earlier try).

Answer 2

Supposing that you have a folder shared between all workers, an alternative solution is using a file to store whatever you need to pass between task runs, including multiple runs of the same one. This solution is a good replacement of XCOM overall, which is usually a more expensive operation (DB) than simple file operations. Just create a subfolder from your DAG run id and use that across all the task of your DAG.