How to convert JSON string column row into a queryable table

Question

I have exported to BigQuery from Firestore a whole collection to perform certain queries on it.

After the data was populated in my BigQuery console, now I can query the whole set like this

SELECT * 
FROM `myapp-1a602.firestore_orders.orders_raw_changelog` 
LIMIT 1000

Now, this statement throws my different columns, but the one I'm looking for is the data column, in my data column is each document JSON, but is in json format and I need to query all this values.

Now, this is the data from one row

{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}

This data is inside the data column, so if I do this

SELECT data 
FROM `myapp-1a602.firestore_orders.orders_raw_changelog` 
LIMIT 1000

I will get all the json values for each document, but I don't know how to query that values, lets say I want to know all orders with status 5 and shopName My awesome shop , now, I need to do something with this json to convert it into a table? does I need to perform the query in the json itself?

How can I query this json output?

Thanks

Answer 1

You canwork with the json functiosn like the

CrEATE Table products (id Integer,attribs_json JSON );

 INSERT INTO products VALUES (1,'{ "cart": [{ "qty": 1, "description": "Sprite 1 L", "productName": "Sprite 1 Liter", "price": 1.99, "productId": 9 }], "storeName": "My awesome shop", "status": 5, "timestamp": { "_seconds": 1590713204, "_nanoseconds": 916000000 } }');

 select * from products where attribs_json->"$.status" = 5 AND attribs_json->"$.storeName" = 'My awesome shop';

 id | attribs_json -: |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 |  {"cart": [{"qty": 1, "price": 1.99, "productId": 9, "description": "Sprite 1 L", "productName": "Sprite 1 Liter"}], "status": 5, "storeName": "My awesome shop", "timestamp": {"_seconds": 1590713204, "_nanoseconds": 916000000}} 

db<>fiddle here

select attribs_json->"$.storeName",attribs_json->"$.status",attribs_json->"$.cart[0].qty" from products where attribs_json->"$.status" = 5 AND attribs_json->"$.storeName" = 'My awesome shop';

 attribs_json->"$.storeName" |  attribs_json->"$.status" |  attribs_json->"$.cart[0].qty":-------------------------- |:----------------------- |:---------------------------- "My awesome shop" |  5 |  1 

db<>fiddle here

And there is JSON_EXTRACT for mysql 5.7 and above.

Finally that is in the end only text, so you could use also REGEXP or RLIKE

To transfer the jaso again to rows, you can use JSON_TABLE

Answer 2

I need to do something with this json to convert it into a table? does I need to perform the query in the json itself?

Below is for BigQuery Standard SQL

#standardSQL
SELECT * EXCEPT(data, cart_item), 
  JSON_EXTRACT(data, '$.status') AS status, 
  JSON_EXTRACT(data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_item, '$.qty') AS qty,
  JSON_EXTRACT(cart_item, '$.description') AS description,
  JSON_EXTRACT(cart_item, '$.productName') AS productName,
  JSON_EXTRACT(cart_item, '$.price') AS price,
  JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item   

If to apply to sample data from your question as in below example

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 order_id, '''
{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    },{
        "qty": 2,
        "description": "Fanta 1 L",
        "productName": "Fanta 1 Liter",
        "price": 1.99,
        "productId": 10
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}  
'''  data 
)
SELECT * EXCEPT(data, cart_item), 
  JSON_EXTRACT(data, '$.status') AS status, 
  JSON_EXTRACT(data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_item, '$.qty') AS qty,
  JSON_EXTRACT(cart_item, '$.description') AS description,
  JSON_EXTRACT(cart_item, '$.productName') AS productName,
  JSON_EXTRACT(cart_item, '$.price') AS price,
  JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item   

result is

Row order_id    status  storeName           qty     description     productName         price   productId    
1   1           5       "My awesome shop"   1       "Sprite 1 L"    "Sprite 1 Liter"    1.99    9    
2   1           5       "My awesome shop"   2       "Fanta 1 L"     "Fanta 1 Liter"     1.99    10   

Answer 3

What you must do is to extract the values from the json data as: SELECT....... WHERE data->'$.storeName'= "My awesome shop" and data->'$.status' = 5

Extracting from the 'cart' or ´the 'timestamp' keys will give you a Json object that needs further extracting to get the data. I hope it'll help you You probably want to have a look at the MySql documentation ( https://dev.mysql.com/doc/refman/8.0/en/json.html ) or https://www.mysqltutorial.org/mysql-json/ .

Answer 4

You can use UNNEST in the WHERE clause to access the cart's columns, and JSON_EXTRACT functions in the WHERE clause to filter the rows wanted. You need to take care on accessing either the json root or the array cart ; json_data and cart_items in the example below (by the way, in your example shopName doesn't exist but storeName does).

WITH
  `myapp-1a602.firestore_orders.orders_raw_changelog` AS (
  SELECT
    '{"cart": [{"qty": 1,"description": "Sprite 1 L","productName": "Sprite 1 Liter","price": 1.99,"productId": 9}, {"qty": 11,"description": "Sprite 11 L","productName": "Sprite 11 Liter","price": 11.99,"productId": 19}],"storeName": "My awesome shop","status": 5,"timestamp": {"_seconds": 1590713204,"_nanoseconds": 916000000}}' json_data )
SELECT
  JSON_EXTRACT(json_data, '$.status') AS status,
  JSON_EXTRACT(json_data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_items, '$.productName') AS product,
  JSON_EXTRACT_SCALAR(cart_items, '$.qty') AS qty
FROM
  `myapp-1a602.firestore_orders.orders_raw_changelog`,
  UNNEST(JSON_EXTRACT_ARRAY(json_data, '$.cart')) AS cart_items
WHERE
  JSON_EXTRACT(json_data,'$.storeName') like "\"My awesome shop\"" AND 
  CAST(JSON_EXTRACT_SCALAR(json_data,'$.status') AS NUMERIC) = 5