在 Airflow 中的任務中訪問 Json 數據

Question

Hi i am triggering a dag with a dag id dag_1 using airflow's REST api and python requests module and in this request I want to send the json data (dic) too. 這就是我和發送的方式

import json
import requests
from requests.auth import HTTPBasicAuth

dic = {
    "flag": "flag",
    "files": "files",
    "upload_path": "config.UPLOAD_FOLDER",
    "tmp_path": "config.TMP_FOLDER",
    "dataset_id": "dataset_id",
    "dicom_meta_data": "dicom_meta_data",
    "user_id": "request.user.id",
    "protocol": "http if request.is_secure() else http",
    "current_site": "request.get_host()",
    "deidentify": "deidentify",
    "email": "request.user.email",
}
json_object = json.dumps(dic)
data = {
    "conf": {},
    "dag_run_id": "trigger_16",
    "logical_date": "2022-08-29T11:33:49.726Z",
}
headers={
    'Content-type':'application/json',
    'Accept':'application/json'
}
json_payload = json.dumps(data)
r = requests.post("http://localhost:8080/api/v1/dags/dag_1/dagRuns", auth=HTTPBasicAuth("airflow", "airflow"), data=json_payload, headers=headers, json=json_object)
print(r.status_code)
print(r.text)

並且工作正常。 但現在我想在 airflow 的第一個任務中訪問這個 json，我無法做到。 任何人都可以幫助我嗎？

謝謝

Answer 1

如果你想將 dict 傳遞給你的 run 以便通過dag_run.conf或params訪問它，你應該將它添加到你的 run 數據中：

import json
import requests
from requests.auth import HTTPBasicAuth

dic = {
    "flag": "flag",
    "files": "files",
    "upload_path": "config.UPLOAD_FOLDER",
    "tmp_path": "config.TMP_FOLDER",
    "dataset_id": "dataset_id",
    "dicom_meta_data": "dicom_meta_data",
    "user_id": "request.user.id",
    "protocol": "http if request.is_secure() else http",
    "current_site": "request.get_host()",
    "deidentify": "deidentify",
    "email": "request.user.email",
}
data = {
    "conf": dic,
    "dag_run_id": "trigger_16",
    "logical_date": "2022-08-29T11:33:49.726Z",
}
headers={
    'Content-type':'application/json',
    'Accept':'application/json'
}
json_payload = json.dumps(data)
r = requests.post("http://localhost:8080/api/v1/dags/dag_1/dagRuns", auth=HTTPBasicAuth("airflow", "airflow"), json=json_payload, headers=headers)
print(r.status_code)
print(r.text)