BigQuery 外部表运算符使用错误的架构路径

Question

Here is a snippet from a DAG that I am working on

create_ext_table = bigquery_operator.BigQueryCreateExternalTableOperator(
    task_id='create_ext_table',
    bucket='bucket-a',
    source_objects='path/*',
    schema_object='bucket-b/data/schema.json',
    destination_project_dataset_table='sandbox.write_to_BQ',
    source_format='CSV',
    field_delimiter=';')

create_ext_table

When I run the code, I am getting the following error on Composer 1.10.10+composer :

404 GET https://storage.googleapis.com/download/storage/v1/b/bucket-a/o/bucket-b%2Fdata%2Fschema.json?alt=media: (u'Request failed with status code', 404, u'Expected one of', 200, 206)

As seen in the error, airflow concat the bucket param with the schema_objet param ... Is there any workaround with this ? Because I cannot store the table schema and the table files in the same bucket

Thanks

Answer 1

This is expected as you can see in the source code for the operator here that we use the bucket argument to get the schema_object , so the operator assumes you have them in the same bucket.

As you mentioned you cannot store them there are a few workarounds that you can try, I'll speak to them at a high level:

1. You can extend the operator and override the execute method in which you retrieve the data from the bucket you care about
2. You can add an upstream task to move the schema object to bucket-a using GoogleCloudStorageToGoogleCloudStorageOperator . This requires handling the schema_object different from the way the source code handles it. Namely parsing it for the bucket name and object path then retrieving it. Alternatively you can create your own argument (something like schema_bucket ) and use it in a similar manner.
   1. You can also delete this object using GoogleCloudStorageDeleteOperator as a downstream task after creating the external table so it does not have to be persisted in `bucket

Final note on the schema_object argument, it's meant to be the GCS path as it uses the same bucket , so if you use the already defined operator it should be schema_object='data/schema.json',