如何在bigquery中将具有不同键数的大型JSON字符串的列展平到表中

Question

I have a google bigquery table with a column containing large JSON strings. In each row, there is a different number of keys and nested keys that I would like to flatten into columns.

My table looks as follows:

id	payload
1	{"key1":{"value":"1"},"key2":2,"key3":1,"key4":"abcde,"version":10}
2	{"key1":{"value":"2"},"key2":5,"key3":2,"key4":"defg,"version":11}

I have managed to extract single columns by using the bq functions JSON_EXTRACT_VALUE and/or JSON_EXTRACT_SCALAR:

SELECT id, JSON_EXTRACT_VALUE(payload, '$.key1') as key1
FROM `project.dataset.table`

etc., however I don't want to hand code more than 100 keys which are nested in the JSON column. There has to be a better way!

I am grateful for any kind of support!

Answer 1

Consider below approach

create temp function  extract_keys(input string) returns array<string> language js as """
  return Object.keys(JSON.parse(input));
  """;
create temp function  extract_values(input string) returns array<string> language js as """
  return Object.values(JSON.parse(input));
  """;
create temp function extract_all_leaves(input string) returns string language js as '''
  function flattenObj(obj, parent = '', res = {}){
    for(let key in obj){
        let propName = parent ? parent + '.' + key : key;
        if(typeof obj[key] == 'object'){
            flattenObj(obj[key], propName, res);
        } else {
            res[propName] = obj[key];
        }
    }
    return JSON.stringify(res);
  }
  return flattenObj(JSON.parse(input));
  ''';

create temp table temp_table as (
  select offset, key, value, id
  from your_table t, 
  unnest([struct(extract_all_leaves(payload) as leaves)]),
  unnest(extract_keys(leaves)) key with offset
  join unnest(extract_values(leaves)) value with offset
  using(offset) 
);  

execute immediate (select '''
  select * from (select * except(offset) from temp_table)
  pivot (any_value(value) for replace(key, '.', '__') in (''' || keys_list || '''
  ))'''
from (select string_agg('"' || replace(key, '.', '__') || '"', ',' order by offset) keys_list from (
  select key, min(offset) as offset from temp_table group by key
))
);     

if applied to sample data as in your question

the output is