[英]BigQuery DeDuplication on two columns as unique key
We use BigQuery religiously and have two tables that essentially were updated in parallel by different process. 我们虔诚地使用BigQuery,并且实际上有两个表是由不同的进程并行更新的。 The problem I have we don't have a unique identifier for tables and the goal is to combine the two tables with zero duplication if possible.. The unique identifier is two columns combined. 我有一个问题,我们没有表的唯一标识符,目标是在可能的情况下将两个表合并为零重复。唯一标识符是将两列组合在一起。
I've tried various MySQL-based queries, but none seem to work in BigQuery. 我已经尝试过各种基于MySQL的查询,但似乎都无法在BigQuery中使用。 So I am posting here for some assistance. 因此,我在这里发布一些帮助。 :) :)
Step 1. Copy the "clean" table into a new merged table. 步骤1.将“ clean”表复制到新的合并表中。
Step 2. Query the "dirty" (old) table and insert any missing entries. 步骤2.查询“ dirty”(旧)表并插入所有丢失的条目。
Query Attempt 1: 查询尝试1:
SELECT
COUNT(c.*)
FROM
[flash-student-96619:device_data.device_datav3_20160530] AS old
WHERE NOT EXISTS (
SELECT
1
FROM
[flash-student-96619:device_data_v7_merged.20160530] AS new
WHERE
new.dsn = old.dsn
AND new.timestamp = old.timestamp
)
Error: error at: 6.1 - 10.65. 错误:错误发生在:6.1-10.65。 Only one query can be executed at a time. 一次只能执行一个查询。
Query Attempt 2: 查询尝试2:
SELECT
*
FROM
[flash-student-96619:device_data.device_datav3_20160530]
WHERE
(dsn, timestamp) NOT IN (
SELECT
dsn,
timestamp
FROM
[flash-student-96619:device_data_v7_merged.20160530]
)
Error: Encountered " "," ", "" at line 6, column 7. Was expecting: ")" ... 错误:在第6行第7列遇到“”,“”,“”。期望:“)” ...
Honestly, if I could do this in one query I would be happy. 老实说,如果我可以在一个查询中做到这一点,我会很高兴。 I need to fetch from two tables, and make a new one with unique data. 我需要从两个表中获取数据,并使用唯一数据创建一个新表。
Any assistance? 有什么帮助吗?
Something like below should work 像下面这样的东西应该工作
SELECT *
FROM (
SELECT *,
ROW_NUMBER() OVER(PARTITION BY dsn, timestamp) AS dup
FROM
[flash-student-96619:device_data.device_datav3_20160530],
[flash-student-96619:device_data_v7_merged.20160530]
)
WHERE dup = 1
I recommend using explicit list of fields instead of * in outer SELECT so you can omit dup from actual output 我建议在外部SELECT中使用显式字段列表而不是*,以便您可以从实际输出中省略dup
A bit late, but I wanted to point out that your original query works with minor modifications using standard SQL (uncheck the "Use Legacy SQL" box under "Show Options"). 有点晚了,但我想指出的是,您的原始查询可以使用标准SQL进行较小的修改(取消选中“显示选项”下的“使用旧版SQL”框)。 I just had to change new
to something else, since that's a reserved keyword. 我只是不得不将new
更改为其他内容,因为这是一个保留关键字。 For example, this query is valid: 例如,此查询有效:
WITH OldData AS (
SELECT
x AS dsn,
TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL x HOUR) AS timestamp
FROM UNNEST([1, 2, 3, 4]) AS x),
NewData AS (
SELECT
x AS dsn,
TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL x HOUR) AS timestamp
FROM UNNEST([5, 2, 1, 6]) AS x)
SELECT
COUNT(*)
FROM OldData oldData
WHERE NOT EXISTS (
SELECT 1
FROM NewData newData
WHERE
newData.dsn = oldData.dsn
AND newData.timestamp = oldData.timestamp
);
+-----+
| f0_ |
+-----+
| 2 |
+-----+
In regard to your second attempt, you can do: 关于第二次尝试,您可以执行以下操作:
WITH OldData AS (
SELECT
x AS dsn,
TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL x HOUR) AS timestamp
FROM UNNEST([1, 2, 3, 4]) AS x),
NewData AS (
SELECT
x AS dsn,
TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL x HOUR) AS timestamp
FROM UNNEST([5, 2, 1, 6]) AS x)
SELECT
*
FROM OldData
WHERE
STRUCT(dsn, timestamp) NOT IN (
SELECT AS STRUCT
dsn,
timestamp
FROM NewData);
+-----+---------------------+
| dsn | timestamp |
+-----+---------------------+
| 3 | 2016-07-21 11:54:08 |
| 4 | 2016-07-21 10:54:08 |
+-----+---------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.