Fetch values from another table for each column written in an array for each ID

Question

I have a dataset that looks like this

sample_data_1

select 'Alice' AS ID, 2 AS col1, 5 AS col2, 6 AS col3, 0 AS col4
union all
select 'Bob' AS ID, 1 AS col1, 4 AS col2, -2 AS col3, 7 AS col4

and a dataset that looks like this. This provides a list of important columns for each ID

sample_data_2

select 'Alice' AS ID, [STRUCT('col1' AS column, 1 AS rank), STRUCT('col4' AS column, 2 AS rank), STRUCT('col3' AS column, 3 AS rank)] AS important_columns
union all
select 'Bob' AS ID, [STRUCT('col4' AS column, 1 AS rank), STRUCT('col2' AS column, 2 AS rank), STRUCT('col1' AS column, 3 AS rank)]

I would like to add the values of the important columns for each ID from sample_data_1

So I can have an output that looks like this

select 'Alice' AS ID, [STRUCT('col1' AS column, 1 AS rank, 2 AS value), STRUCT('col4' AS column, 2 AS rank, 0 AS value), STRUCT('col3' AS column, 3 AS rank, 6 AS value)] AS important_columns
union all
select 'Bob' AS ID, [STRUCT('col4' AS column, 1 AS rank, 7 AS value), STRUCT('col2' AS column, 2 AS rank, 4 AS value), STRUCT('col1' AS column, 3 AS rank, 1 AS value)]

output

I would like the code to be dynamic, so will work even if column names change or number of columns change

Answer 1

Consider below query.

EXECUTE IMMEDIATE FORMAT("""
  CREATE TEMP TABLE sample_data1_unpivoted AS 
  SELECT ID, ARRAY_AGG(STRUCT(column, value)) outputs FROM sample_data1 UNPIVOT (value FOR column IN (%s)) GROUP BY 1
""", ARRAY_TO_STRING(
  REGEXP_EXTRACT_ALL(TO_JSON_STRING((SELECT AS STRUCT * EXCEPT (ID) FROM sample_data1 LIMIT 1)), r'"([^,{]+)":'), ',')
);

SELECT ID, ARRAY (
         SELECT AS STRUCT column, rank, value 
           FROM t1.important_columns LEFT JOIN t2.outputs USING (column)
       ) important_columns
  FROM sample_data2 t1
  LEFT JOIN sample_data1_unpivoted t2 USING (ID);

Query results