Error in Airflow SQLCheckOperator - AttributeError: 'NoneType' object has no attribute 'upper'

Question

I want to check if my table is loaded correctly or not. If it is not loaded correctly then the number of records will be zero. I am using SQLCheckOperator to do this task.

This is the code

from airflow.operators.sql import SQLCheckOperator
from datetime import date, timedelta

CURRENT_DATE = str(date.today() - timedelta(2))

TABLE_NAME = "foo"
search_monolith_post_sanity = SQLCheckOperator(
    task_id="search_monolith_post_sanity",
    sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
    bigquery_conn_id='bigquery_default',
    use_legacy_sql=False,
    dag=dag
)

I got the below error:

Executing SQL check: SELECT COUNT(*) FROM `foo` WHERE feed_date = DATE_SUB('2021-01-31', INTERVAL 1 DAY)
[2021-02-02 07:16:43,664] {taskinstance.py:1153} ERROR - 'NoneType' object has no attribute 'upper'
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 986, in _run_raw_tas
result = task_copy.execute(context=context
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 95, in execut
  records = self.get_db_hook().get_first(self.sql
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 116, in get_db_hoo
  return BaseHook.get_hook(conn_id=self.conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 94, in get_hoo
  connection = cls.get_connection(conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 87, in get_connectio
  conn = random.choice(list(cls.get_connections(conn_id))
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 83, in get_connection
  return secrets.get_connections(conn_id
File "/usr/local/lib/airflow/airflow/secrets/__init__.py", line 55, in get_connection
  conn_list = secrets_backend.get_connections(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/base_secrets.py", line 64, in get_connection
  conn_uri = self.get_conn_uri(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/environment_variables.py", line 39, in get_conn_ur
  environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper

I have tried using BigQueryCheckOperator and CheckOperator instead of SQLCheckOperator but ran into error. If I replace BigQueryCheckOperator with BigQueryOperator the code works fine and I get zero as output.

I am new to airflow. Any help is much appreciated. Thanks !!

Answer 1

Assumption : You are using Airflow >= 2.0.0

Use the following code, notice usage of BigQueryCheckOperator and that I used gcp_conn_id instead of bigquery_conn_id .

from airflow.providers.google.cloud.operators.bigquery import BigQueryCheckOperator
from datetime import date, timedelta

CURRENT_DATE = str(date.today() - timedelta(2))

TABLE_NAME = "foo"
search_monolith_post_sanity = BigQueryCheckOperator(
    task_id="search_monolith_post_sanity",
    sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
    gcp_conn_id='bigquery_default',
    use_legacy_sql=False,
    dag=dag
)

Answer 2

If you look at the line before the error message in the stacktrace.

environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper'

In this case the NoneType object that is having upper() called on it is conn_id .

If you're using Airflow 1.10.15 the documentation for this operator has a rather important Note buried at the bottom

Note that this is an abstract class and get_db_hook needs to be defined. Whereas a get_db_hook is hook that gets a single record from an external source.

Also note that the definition of the function appears to expect a conn_id parameter.