[英]Error in Airflow SQLCheckOperator - AttributeError: 'NoneType' object has no attribute 'upper'
I want to check if my table is loaded correctly or not.我想检查我的表是否正确加载。 If it is not loaded correctly then the number of records will be zero.如果未正确加载,则记录数将为零。 I am using SQLCheckOperator to do this task.我正在使用 SQLCheckOperator 来执行此任务。
This is the code这是代码
from airflow.operators.sql import SQLCheckOperator
from datetime import date, timedelta
CURRENT_DATE = str(date.today() - timedelta(2))
TABLE_NAME = "foo"
search_monolith_post_sanity = SQLCheckOperator(
task_id="search_monolith_post_sanity",
sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
bigquery_conn_id='bigquery_default',
use_legacy_sql=False,
dag=dag
)
I got the below error:我收到以下错误:
Executing SQL check: SELECT COUNT(*) FROM `foo` WHERE feed_date = DATE_SUB('2021-01-31', INTERVAL 1 DAY)
[2021-02-02 07:16:43,664] {taskinstance.py:1153} ERROR - 'NoneType' object has no attribute 'upper'
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 986, in _run_raw_tas
result = task_copy.execute(context=context
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 95, in execut
records = self.get_db_hook().get_first(self.sql
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 116, in get_db_hoo
return BaseHook.get_hook(conn_id=self.conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 94, in get_hoo
connection = cls.get_connection(conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 87, in get_connectio
conn = random.choice(list(cls.get_connections(conn_id))
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 83, in get_connection
return secrets.get_connections(conn_id
File "/usr/local/lib/airflow/airflow/secrets/__init__.py", line 55, in get_connection
conn_list = secrets_backend.get_connections(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/base_secrets.py", line 64, in get_connection
conn_uri = self.get_conn_uri(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/environment_variables.py", line 39, in get_conn_ur
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper
I have tried using BigQueryCheckOperator and CheckOperator instead of SQLCheckOperator but ran into error.我尝试使用 BigQueryCheckOperator 和 CheckOperator 而不是 SQLCheckOperator 但遇到了错误。 If I replace BigQueryCheckOperator with BigQueryOperator the code works fine and I get zero as output.如果我用 BigQueryOperator 替换 BigQueryCheckOperator,则代码可以正常工作,并且我得到零作为 output。
I am new to airflow.我是 airflow 的新手。 Any help is much appreciated.任何帮助深表感谢。 Thanks !!谢谢 !!
Assumption : You are using Airflow >= 2.0.0假设:您正在使用 Airflow >= 2.0.0
Use the following code, notice usage of BigQueryCheckOperator
and that I used gcp_conn_id
instead of bigquery_conn_id
.使用以下代码,注意BigQueryCheckOperator
的用法,并且我使用gcp_conn_id
而不是bigquery_conn_id
。
from airflow.providers.google.cloud.operators.bigquery import BigQueryCheckOperator
from datetime import date, timedelta
CURRENT_DATE = str(date.today() - timedelta(2))
TABLE_NAME = "foo"
search_monolith_post_sanity = BigQueryCheckOperator(
task_id="search_monolith_post_sanity",
sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
gcp_conn_id='bigquery_default',
use_legacy_sql=False,
dag=dag
)
If you look at the line before the error message in the stacktrace.如果您查看堆栈跟踪中错误消息之前的行。
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper'
In this case the NoneType
object that is having upper()
called on it is conn_id
.在这种情况下,调用了upper()
的NoneType
object 是conn_id
。
If you're using Airflow 1.10.15 the documentation for this operator has a rather important Note buried at the bottom如果您使用的是 Airflow 1.10.15,则此运算符的文档在底部有一个相当重要的注释
Note that this is an abstract class and get_db_hook needs to be defined.请注意,这是一个抽象的 class 并且需要定义 get_db_hook。 Whereas a get_db_hook is hook that gets a single record from an external source.而 get_db_hook 是从外部源获取单个记录的钩子。
Also note that the definition of the function appears to expect a conn_id
parameter.另请注意,function 的定义似乎需要一个conn_id
参数。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.