[英]How to decode airflow table dag_run column conf value
How do i decode the value of airflow table dag_run column conf value so I can read the value of conf column from the table我如何解码气流表 dag_run 列 conf 值的值,以便我可以从表中读取 conf 列的值
Thanks谢谢
I recently ran into the same thing, and it turns out the conf
column is a binary Pickle string.我最近遇到了同样的事情,结果
conf
列是一个二进制 Pickle 字符串。
I found the answer in the Airflow internal module documentation https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dagrun.html#DagRun.conf我在 Airflow 内部模块文档https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dagrun.html#DagRun.conf中找到了答案
import pickle
# function to return SQLAlchemy Engine connected to Airflow
engine = get_airflow_db_engine()
# query and convert conf
query = """
select dag_id, execution_date, state, run_id, conf
from dag_run
limit 25
"""
df = pd.read_sql(query, engine)
df['conf'] = df['conf'].apply(lambda x: pickle.loads(x))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.