变体列中唯一元素的雪花查询性能

Question

I am querying a Snowflake view that contains many TB of semi-structured json data. When I query the variant column of interest for an element that is not unique among the records, results are returned within seconds:

SELECT json_data:element1 FROM table WHERE json_data:common_category = 'CATEGORY1';

When I query the variant column of interest for an element that is unique among the records, runtime slows to some unacceptable amount of time that I have not yet reached:

SELECT json_data:element1 FROM table WHERE json_data:unique_id = 'ID123456';

I believe that flattening the unique element into a relational form outside of the variant column would increase performance, but I am not a DBA with these permissions. Is there some way to tune my query such that looking up a single record based on the variant column json data will yield acceptable performance?

Answer 1

Snowflake stores internally variant (json) data, in a independent column like structure for the 100+ most common elements, and the remainder in a leftovers like column. Those virtual columns have min/max, distribution like stats like normal columns have.

notes 1 notes 2

This means on the major columns of your data, they can prune lots of unneeded partitions for reading (if your data is naturally ordered in a way the helps this).

It also means if your are using a couple of columns from the JSON it reads just those stripes, thus less IO.

Also when you select the whole blob like you do here, the second point does not come into play, as the READ for the SELECT and the READ for the WHERE are the same.

So for you queries you will see the first query all a small number of partition in. For your second query you will see it plans to read all partitons.

If you alter your first query to:

SELECT json_data:common_category FROM table WHERE json_data:common_category = 'CATEGORY1';

You will see that the number of partition read is the same as the first example, but the number of bytes read should be a fraction.

Again like for normal tables you should always name all your columns and avoid SELECT * FROM TABLE as then the plan knows what to pull. You will see a statistically faster compile times when you name all your first order columns and all your variant columns.

In the context of making it faster:

if you have to have All the JSON columns then and

SELECT json_data FROM table WHERE json_data:common_category = 'CATEGORY1';

has acceptable speed then do:

SELECT json_data:common_category FROM table WHERE json_data:unique_id = 'ID123456';
SELECT json_data FROM table WHERE json_data:common_category = <answer from prior> and json_data:unique_id = 'ID123456';

this way the first query is reading the least amount form all partitions, and the second is reading the all from the partition that have to be read from..

Now this will not always work if for example the common_category for unique_id = 'ID123456' is common to all partitions, but if you have some other column on all rows, that is sequential or aligned with the sort of the data (be that how you ingest the data thus the write order, or how you order the data if you have it clustered). Then select the filter column and the order columns then select the full match with the focusing effect of the ordering column.

We have very similar audit data the above pattern is used on, and other data that we store in multiple tables, some of the tables are super skinning and ordered (via cluster keys) and then we have a key that is insert_time of both that fast table and a wide/fat json table with all the "extra's" that are often not used, but are written in _insert_time order, thus finding the desired data in the fast table allow read the wide table with reduced partitions.