在 Big Query 中以动态方式将 JSON 拆分为列

Question

I have the following JSON:

{
    "rewards": {
        "reward_1": {
            "type": "type 1",
            "amount": "amount 1"
        },
        "reward_2": {
            "type": "type 2",
            "amount": "amount 2"
        },
        "reward_3": {
            "type": "type 3",
            "amount": "amount 3"
        },
        "reward_4": {
            "type": "type 4",
            "amount": "amount 4"
        }
    }
}

This JSON is dynamic and I don't necessarily know how many rewards it will get, here it's 4 but it can be 2 or 8 etc.

I want to write a query in Big Query that will parse those values dynamically without knowing how many of them exist, and then split them into column, like this:

Thank you!

Answer 1

Hope these are helpful.

1. since a JSON data is dynamic, first step is to find a max reward sequence. (I've used a regular expression and max_reward UDF.)
2. and then, extract each reward from a json rewards field in an iterative way.
3. lastly, make the result to be a wide form using PIVOT query.

If you want a more generic solution, you need to use BigQuery dynamic SQL to generate PIVOT columns. I've hard-coded them in the query.

('reward_1', 'reward_2', 'reward_3', 'reward_4')

query:

CREATE TEMP TABLE sample AS 
SELECT 1 AS id, '{"rewards": { "reward_1": { ... ' AS json -- put your json here
 UNION ALL
SELECT 2 AS id, '{"rewards": { "reward_1": { ... ' AS json -- put your another json here
;

CREATE TEMP FUNCTION extract_reward(json STRING, seq INT64)
RETURNS STRUCT<type STRING, amount STRING>
LANGUAGE js AS """
  return JSON.parse(json)['reward_' + seq];
""";

CREATE TEMP FUNCTION max_reward(arr ARRAY<STRING>) AS ((
  SELECT MAX(CAST(v AS INT64)) FROM UNNEST(arr) v
));

SELECT * FROM (
  SELECT id,
         'reward_' || seq AS reward, 
         extract_reward(FORMAT('%t', JSON_QUERY(json, '$.rewards')), seq) AS value
    FROM sample, UNNEST(GENERATE_ARRAY(1, max_reward(REGEXP_EXTRACT_ALL(json, r'"reward_([0-9]+)"')))) seq
)  PIVOT (ANY_VALUE(value) FOR reward IN ('reward_1', 'reward_2', 'reward_3', 'reward_4'));

output:

▶ Split a reward STRUCT column into separate columns

SELECT * FROM (
  SELECT id,
         'reward_' || seq || '_' || IF (offset = 0, 'type', 'amount') AS reward,
         value
    FROM sample,
         UNNEST(GENERATE_ARRAY(1, max_reward(REGEXP_EXTRACT_ALL(json, r'"reward_([0-9]+)"')))) seq, 
         UNNEST([extract_reward(FORMAT('%t', JSON_QUERY(json, '$.rewards')), seq)]) pair,
         UNNEST([pair.type, pair.amount]) value WITH OFFSET
)  PIVOT (ANY_VALUE(value) FOR reward IN ('reward_1_type', 'reward_2_type', 'reward_3_type', 'reward_4_type', 'reward_1_amount', 'reward_2_amount', 'reward_3_amount', 'reward_4_amount'));

output: