在 GCP 中创建带分区的外部表

Question

I am trying to creating an external table with Partition below is the reference image i am using.

Here is what i am intending to do:

I have files flowing into this folder:

I need to query the external table based on the date:

eg:

select * from where _PartitionDate ='';

My specific query is what should i fill in the GCS bucket & source Data partitioning fields.

Thank you.

Answer 1

According to the documentation that Guillaume provided [1], you should click on the Source data partitioning box and provide the following link there:

gs://datalake-confidential-redacted/ExternalTable_Data/

Also, the Table type should be External table .

Once that is fixed, you should be able to create the table. I have reproduced the issue on my own and it is working.

[1] - https://cloud.google.com/bigquery/docs/hive-partitioned-queries-gcs#hive-partitioning-options

Answer 2

This part of the documentation should help you. You need to check the Source data partitioning and then to fill in your prefix URI such as

gs://datalake-confidential-redacted/ExternalTable_Data/{dt:DATE}

And then, use this dt field as any field in your queries

SELECT *
FROM `externale-table`
WHERE dt = "2020-01-10"

Answer 3

Custom Wizard has an issue with this approch. Once we used Teraform scripts it has been successful. It mandates a need to mark HIVE partition to custom & once the date column is created it is added as column into the table. there by allowing to query.