使用 HttpSensor 触发气流 DAG

Question

I'm trying to implement the exact use case as mentioned here , but I'm having issues. I can't seem to actually set the trigger off in order to execute task t1 , even though I'm changing the google sheet referenced. I've made sure all path references are right and I have access to the necessary APIs (google drive). I'm very new to Airflow, so I suspect I'm missing fundamental in my dag:

from airflow import DAG
from airflow.sensors.http_sensor import HttpSensor
from airflow.operators.bash_operator import BashOperator
from airflow.contrib.operators.bigquery_operator import BigQueryOperator
from datetime import timedelta, datetime

args = {
    'owner': 'airflow',
    'depends_on_past': False, 
    'start_date': datetime(2020, 11, 23),
    'email': 'my_email@gmail.com',
    'email_on_failure': True,
}

dag = DAG(
    'my_dag',
    default_args=args,
    description='blah blah',
    schedule_interval='*/5 * * * *'
)

# sensor to detect changes in google sheet
s = HttpSensor(
    task_id='check_sheet_change',
    http_conn_id='http_default',
   endpoint='https://www.googleapis.com/drive/v3/files/hiddenfromyouid',
    request_params={},
    response_check=lambda response: response.json()['modifiedTime'] > (datetime.now() - timedelta(seconds=30)),
    poke_interval=30,
    dag=dag,
)

# if s is triggered, do stuff in python
t1 = BashOperator(
    task_id='do stuff in python',
    bash_command='python3 /home/airflow/gcs/dags/pythons/mypyfile.py',
    dag=dag
)

s >> t1

What am I missing? Any help is appreciated.

Answer 1

endpoint of HttpSensor is for relative part of URL.

try this, and http_default connection should direct https://www.googleapis.com

s = HttpSensor(
    task_id='check_sheet_change',
    http_conn_id='http_default',
   endpoint='/drive/v3/files/hiddenfromyouid',
    request_params={},
    response_check=lambda response: response.json()['modifiedTime'] > (datetime.now() - timedelta(seconds=30)),
    poke_interval=30,
    dag=dag,
)