BigQuery: Loop through tables in dataset and drop tables with a specific prefix

Question

Our development environment in BigQuery is isolated to a development dataset ie dev in BigQuery. The environments are further isolated by a prefix of the ticket for each table ie DATA-100-change-table would correspond to the DATA-100 ticket.

I am aware of setting TTL for BigQuery , however, I am also interested in having a query I could run by hand to delete the tables.

So far, I have the below:

begin
  -- Create temporary table of tables to drop from `dev` with prefix
  create temp table to_drop (drop_string STRING)
  as
  (
    select concat("drop table if exists ", table_catalog, ".", table_schema, ".", table_name)
    from dev.INFORMATION_SCHEMA.TABLES
    where table_name like "DATA-100%"
  );

  -- Loop through table and execute drop_string statements
  for drop_statement in (select drop_string from to_drop)
  do
    execute immediate drop_statement;
  end for;
end

However, this fails with the following error:

Query error: Cannot coerce expression drop_statement to type STRING at [14:23]

Is my approach right here? How do I best delete all tables with a prefix in BigQuery?

Also, if possible, I would like this query to handle views as well.

Answer 1

The variable drop_statement in the for loop contains a struct. So you have to access the string with drop_statement.drop_string .

begin
  -- Create temporary table of tables to drop from `dev` with prefix
  create temp table to_drop (drop_string STRING)
  as
  (
    select concat("drop table if exists ", table_catalog, ".", table_schema, ".", table_name)
    from dev.INFORMATION_SCHEMA.TABLES
    where table_name like "DATA-100%" and table_type = 'BASE TABLE'
  );

  -- Loop through table and execute drop_string statements
  for drop_statement in (select drop_string from to_drop)
  do
    execute immediate drop_statement.drop_string;
  end for;
end

To drop VIEWs, just replace table_type = "BASE TABLE" with table_type = "VIEW" and use drop view if exists instead.

Answer 2

Using p13rr0m's answer as inspiration, I've come up with the below:

begin
  create temp table to_drop (drop_string STRING)
  as 
  (
    select
      case 
        when table_type = "BASE TABLE" then concat("drop table if exists `", table_catalog, ".", table_schema, ".", table_name, "`")
        when table_type = "VIEW" then concat("drop view if exists `", table_catalog, ".", table_schema, ".", table_name, "`")
      end as drop_string
    from
      `dev.INFORMATION_SCHEMA.TABLES`
    where
      table_name like "DATA_100%"
  );

  for drop_statement in (select drop_string from to_drop)
  do
    execute immediate drop_statement.drop_string;
  end for;
end

Had to add backticks to the tables as it was failing on dropping views.