使用 CTE 与 SubQuery 重构 SQL 查询

Question

I am creating a dataset from an S3 Bucket, and currently I am trying to improve the performance of the query as the current two approaches I have work but I would like to see a better query and learn how to improve my sql skills. Sorry for no sample dataset to work with as I have not figured out a practical way to provide mock data when pulling from .json files in S3.

QUERY # 1

 WITH block_1 AS
    (
    SELECT 
    VALUE:COL1 AS COL1, 
    VALUE:COL2 AS COL2, 
    VALUE:COL3 AS COL3,
    VALUE:COL4 AS COL4
    from '@S3_BUCKET/', 
     lateral flatten( input => $1:value)), block_2 as 

(
SELECT 
VALUE:COL1 AS COL1, 
max(VALUE:COL4) AS MaxCOL4
from '@S3_BUCKET/', 
lateral flatten( input => $1:value)
group by COL1
 )

select b.COL1 as COL1B, b.COLB as COL1B, 
 a.COL3, a.COL4 from block_1 as A
join block_2 b 
on a.COL1 = b.COL1  and a.COL4 = b.MaxCOL4
 ;

QUERY #2 , I felt was an improvement, especially because you do not need to specify the column you want in the final SELECT statement (as I did above)

select a.* from 
(
SELECT 
VALUE:COL1 AS COL1, 
VALUE:COL2 AS COL2, 
VALUE:COL3 AS COL3,
VALUE:COL4 AS COL4
from '@S3_BUCKET/', 
lateral flatten( input => $1:value))a 
join 
(
select COL1, MAX(COL4) COL4
from 
(
SELECT 
VALUE:COL1 AS COL1, 
VALUE:COL2 AS COL2, 
VALUE:COL3 AS COL3,
VALUE:COL4 AS COL4
from '@S3_BUCKET/', 
 lateral flatten( input => $1:value))
group by COL1) b
on a.COL1 = b.COL1 and a.COL4 = b.Col4;

The two above are my current attempt, wondering if there would be a way to make this query better? The other route I was thinking was possibly using "where in" , and the list of COL1, but essentially then I still have to hit s3 2x , as the queries above.

Answer 1

You should be able to use window functions , specifically RANK() to simplify this query:

WITH block_1 AS (
    SELECT 
    VALUE:COL1 AS COL1, 
    VALUE:COL2 AS COL2, 
    VALUE:COL3 AS COL3,
    VALUE:COL4 AS COL4,
    RANK() OVER (PARTITION BY VALUE:COL1 ORDER BY VALUE:COL4 DESC) AS rk
    FROM '@S3_BUCKET/', 
     lateral flatten( input => $1:value)
)
SELECT COL1, COL2, COL3, COL4
FROM block_1
WHERE rk = 1

This can be simplified thanks to Snowflake's QUALIFY clause, which allows you to use an alias for a window function in what is effectively a HAVING clause:

SELECT 
    VALUE:COL1 AS COL1, 
    VALUE:COL2 AS COL2, 
    VALUE:COL3 AS COL3,
    VALUE:COL4 AS COL4,
    RANK() OVER (PARTITION BY VALUE:COL1 ORDER BY VALUE:COL4 DESC) AS rk
FROM '@S3_BUCKET/', 
     lateral flatten( input => $1:value)
QUALIFY rk = 1

Answer 2

@nick. Use qualify , this will act as where filter and set = 1. Also replace rank with row_number. Does that make sense ?