BigQuery: Select column if it exists, else put NULL?

Question

I am updating a daily dashboard. Assume everyday I have two tables TODAY and BEFORE_TODAY . What I have been doing daily was something like:

SELECT a, b FROM TODAY
UNION ALL
SELECT a,b FROM BEFORE_TODAY;

TODAY table is generated daily and is appended to all the data before it. Now, I need to generate a new column say c and in order to UNION ALL the two, I need that to be available on BEFORE_TODAY as well.

How can I add a conditional statement to BEFORE_TODAY to check if I have a c column and use that column, else use NULL instead of it.

Answer 1

Something is wrong with your data model. You should be putting the data into separate partitions of the same table. Then you would have no problems. You could just query the master table for the partitions you want. The column would appear in the history, with a NULL value.

That is the right solution. You can create a hacked solution, assuming that your tables have a primary key. For instance, if a, b is unique on each row, you could do:

select t.a, t.b, t.c
from today t
union all
select b.a, b.b,
       (select c  -- not qualified on purpose
        from before_today b2
        where b2.a = b.a and b2.b = b.b
       ) as 
from before_today b cross join
     (select null as c) c;

If b.c does not exist, then the subquery returns c.c . If it does exist this it returns b2.c from the subquery.

Answer 2

Although by combining dynamic SQL and INFORMATION_SCHEMA, you can write a script to achieve what you told, this doesn't seems to be the right thing to do.

As part of your data model planning, you should add column c to BEFORE_TODAY ahead of time. The newly added column on existing rows will always have NULL value. Then you can add column to TODAY and reference column c as normal.