在 BigQuery 中，删除除一列之外的每一列上重复的行

Question

在使用多列删除 bigquery 中的重复行时，一种常见的解决方案是使用row_number()并按要删除的多列进行分区。 在我们的情况下，我们有一个宽表（30 列），并希望根据其中 29 列的唯一性删除重复项：

with
  t1 as (
    select 1 as a, 2 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i union all
    select 2 as a, 3 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i union all
    select 3 as a, 4 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i union all
    select 4 as a, 5 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i union all
    select 5 as a, 6 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i union all
    select 6 as a, 2 as b, 3 as c, 4 as d, 5 as e, 6 as f, 7 as g, 8 as h, 9 as i
  )

在上表中，我们要删除除a列之外的所有列的重复项。 因此，第 1 行和第 6 行是重复的，我们要删除其中任何一个，最好删除a列中具有较高值的行，即本例中的第 6 行。 是否可以不使用row_number() over (partition by b,c,d,e,f,g,h,i,...)

Answer 1

您可以考虑以下查询

 SELECT * 
   FROM t1 
QUALIFY ROW_NUMBER() OVER (
          PARTITION BY TO_JSON_STRING((SELECT AS STRUCT t1.* EXCEPT(a)))
              ORDER BY a ASC
        ) = 1;

Answer 2

另外的选择

select any_value(t).* replace(max(a) as a)
from your_table t
group by to_json_string((select as struct * except(a) from unnest([t])))    

与 output