Daily I’m receiving a new table in the BigQuery, I want concatenate this new table data to the main table, dataset schema are same

Question

Daily I'm receiving a new table (example:tablename_20220811) in the BigQuery, I want concatenate this new table data to the main_table, dataset schema are same.

I tried using wild cards,I don't know how to pull the daily loaded table.

Answer 1

You can use BigQuery scheduled queries with an interval (cron) in the schedule parameters:

Example with gcloud cli:

bq query \
  --use_legacy_sql=false \
  --destination_table=mydataset.desttable \
  --display_name='My Scheduled Query' \
  --schedule='every 24 hours' \
  --append_table=true \
  'SELECT
    1
   FROM
    mydataset.tablename_*
    where _TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', CURRENT_DATE())'

In order to target on the expected table, I used a wildcard and a filter based on the table suffix. The table suffix should be equals to the current date as STRING with the following format yyyymmdd .

The cron plan to run the query every day.

You can also configure it directly with the Google Cloud console.

Answer 2

It sounds like you have the right naming format for BigQuery to treat your tables as a single 'date-sharded table'.

You need to ensure that the daily tables

• have the same schema
• are in the same dataset
• have the same name apart from the _yyyymmdd suffix

You will know if this worked because only one table will appear (with an icon showing multiple tables, rather than the usual icon).

With this in hand, you can write queries like

    SELECT
        fieldA,
        fieldB,
      FROM
       `some_dataset.tablename_*`
     WHERE
        _table_suffix BETWEEN '20220101' AND '20221201'

This gives you some idea of what's possible:

• select from the full date-sharded table using backticks (essential!) and the wildcard syntax
• filter using the special _table_suffix meta-field