如何在另一个任务气流中使用查询（bigquery operator）的结果

Question

I have a project in google composser that aims to submit on a daily basis. The code below does that, it works fine.

with models.DAG('reporte_prueba',
    schedule_interval=datetime.timedelta(weeks=4),
    default_args=default_dag_args) as dag:

    make_bq_dataset = bash_operator.BashOperator(
        task_id='make_bq_dataset',
        # Executing 'bq' command requires Google Cloud SDK which comes
        # preinstalled in Cloud Composer.
        bash_command='bq ls {} || bq mk {}'.format(
            bq_dataset_name, bq_dataset_name))
        
    bq_audit_query = bigquery_operator.BigQueryOperator(
        task_id='bq_audit_query',
        sql=query_sql,
        use_legacy_sql=False,
        destination_dataset_table=bq_destination_table_name)

    export_audits_to_gcs = bigquery_to_gcs.BigQueryToCloudStorageOperator(
        task_id='export_audits_to_gcs',
        source_project_dataset_table=bq_destination_table_name,
        destination_cloud_storage_uris=[output_file],
        export_format='CSV')
    
    download_file = GCSToLocalFilesystemOperator(
        task_id="download_file",
        object_name='audits.csv',
        bucket='bucket-reportes',
        filename='/home/airflow/gcs/data/audits.csv',
    )
    email_summary = email_operator.EmailOperator(
        task_id='email_summary',
        to=['aa@bb.cl'],
        subject="""Reporte de Auditorías Diarias 
        Institución: {institution_report} día {date_report}
        """.format(date_report=date,institution_report=institution),
        html_content="""
        Sres.
        <br>
        Adjunto enviamos archivo con Reporte Transacciones Diarias.
        <br>
        """,
        files=['/home/airflow/gcs/data/audits.csv'])

    delete_bq_table = bash_operator.BashOperator(
        task_id='delete_bq_table',
        bash_command='bq rm -f %s' % bq_destination_table_name,
        trigger_rule=trigger_rule.TriggerRule.ALL_DONE)


    (
        make_bq_dataset 
        >> bq_audit_query 
        >> export_audits_to_gcs 
        >> delete_bq_table
    )
    export_audits_to_gcs >> download_file >> email_summary

With this code, I create a table (which is later deleted) with the data that I need to send, then I pass that table to storage as a csv. then I download the.csv to the local airflow directory to send it by mail.

The question I have is that if I can avoid the part of creating the table and taking it to storage. since I don't need it.

for example, execute the query with BigqueryOperator and access the result in ariflow, thereby generating the csv locally and then sending it.

I have the way to generate the CSV but my biggest doubt is how (if it is possible) to access the result of the query or pass the result to another airflow task

Answer 1

Though I wouldn't recommend passing results of sql queries across tasks, XComs in airflow are generally used for the communication between tasks.

https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html

Also you need to create a custom operator to return query results, as I "believe" BigQueryOperator doesn't return query results.