BigQuery循环从另一个表中注册的动态table_names中选择值

Question

I'm looking for a solution to extract data from multiple tables and insert it into another table automatically running a single script. I need to query many tables, so I want to make a loop to select from those table's names dynamically.

I wonder if I could have a table with table names, and execute a loop like:

foreach(i in table_names)
    insert into aggregated_table select * from table_names[i]
end

Answer 1

Below is for BigQuery Standard SQL

#standardSQL
SELECT * FROM `project.dataset1.*`
WHERE _TABLE_SUFFIX IN (SELECT table_name FROM `project.dataset2.list`)

This approach will work if below conditions are met

• all to be processed table from list have exact same schema
• one of those tables is the most recent table - this table will define schema that will be used for all the rest tables in the list
• to meet above bullet - ideally list should be hosted in another dataset

Obviously, you can add INSERT INTO ... to insert result into whatever destination is to be

Please note: Filters on _TABLE_SUFFIX that include subqueries cannot be used to limit the number of tables scanned for a wildcard table, so make sure your are using longest possible prefix - for example

#standardSQL
SELECT * FROM `project.dataset1.source_table_*`
WHERE _TABLE_SUFFIX IN (SELECT table_name FROM `project.dataset2.list`)   

So, again - even though you will select data from specific tables (set in project.dataset2.list ) the cost will be for scanning all tables that match project.dataset1.source_table_* woldcard

While above is purely in BigQuery SQL - you can use any client of your choice to script exacly the logic you need - read table names from list table and then select and insert in loop - this option is simplest and most optimal I think