BigQuery：如何在窗口函數上合並HLL草圖？ （在滾動窗口中計數不同的值）

Question

相關表格示例示例：

+---------------------------+-------------------+
| activity_date - TIMESTAMP | user_id - STRING  |
+---------------------------+-------------------+
| 2017-02-22 17:36:08 UTC   | fake_id_i24385787 |
+---------------------------+-------------------+
| 2017-02-22 04:27:08 UTC   | fake_id_234885747 |
+---------------------------+-------------------+
| 2017-02-22 08:36:08 UTC   | fake_id_i24385787 |
+---------------------------+-------------------+

我需要在一個滾動的時間段內（90天）計算一個大型數據集上活躍的獨立用戶的數量，並且由於數據集的大小而出現問題。

首先，我嘗試使用窗口函數，類似於此處的答案。 https://stackoverflow.com/a/27574474

WITH
  daily AS (
  SELECT
    DATE(activity_date) day,
    user_id
  FROM
    `fake-table`)
SELECT
  day,
  SUM(APPROX_COUNT_DISTINCT(user_id)) OVER (ORDER BY day ROWS BETWEEN 89 PRECEDING AND CURRENT ROW) ninty_day_window_apprx
FROM
  daily
GROUP BY
  1
ORDER BY
  1 DESC

但是，這導致每天獲得不同數量的用戶，然后將這些數量進行匯總-但是，如果不同用戶出現多次，則它們可以在窗口中重復。 因此，這並不是對90天內不同用戶的真實准確衡量。

我嘗試的下一件事是使用以下解決方案https://stackoverflow.com/a/47659590-將每個窗口的所有不同的user_id連接到一個數組，然后計算其中的不同。

WITH daily AS (
  SELECT date(activity_date) day, STRING_AGG(DISTINCT user_id) users
  FROM `fake-table`  
  GROUP BY day
), temp2 AS (
  SELECT
    day, 
    STRING_AGG(users) OVER(ORDER BY UNIX_DATE(day) RANGE BETWEEN 89 PRECEDING AND CURRENT ROW) users
  FROM daily
)

SELECT day, 
  (SELECT APPROX_COUNT_DISTINCT(id) FROM UNNEST(SPLIT(users)) AS id) Unique90Days
FROM temp2

order by 1 desc

但是，這很快就用光了所有大容量的內存。

接下來是使用HLL草圖以較小的值表示不同的ID，因此內存將不再是問題。 我以為我的問題已解決，但是運行以下命令時出現錯誤：錯誤僅是“不支持MERGE_PARTIAL函數”。 我也嘗試了MERGE，並遇到了相同的錯誤。 僅在使用窗口功能時發生。 為每天的價值創建草圖效果很好。

我通讀了BigQuery Standard SQL文檔，但沒有看到關於帶有窗口函數的HLL_COUNT.MERGE_PARTIAL和HLL_COUNT.MERGE的任何信息。 大概應該采用90個草圖並將它們組合為一個HLL草圖，代表90個原始草圖之間的不同值？

WITH
  daily AS (
  SELECT
    DATE(activity_date) day,
    HLL_COUNT.INIT(user_id) sketch
  FROM
    `fake-table`
  GROUP BY
    1
  ORDER BY
    1 DESC),

  rolling AS (
  SELECT
    day,
    HLL_COUNT.MERGE_PARTIAL(sketch) OVER (ORDER BY UNIX_DATE(day) RANGE BETWEEN 89 PRECEDING AND CURRENT ROW) rolling_sketch
    FROM daily)

SELECT
  day,
  HLL_COUNT.EXTRACT(rolling_sketch)
FROM
  rolling
ORDER BY
  1 

“錯誤圖像-不支持功能MERGE_PARTIAL”

任何想法為什么會發生此錯誤或如何調整？

Answer 1

以下是適用於BigQuery Standard SQL的信息，它確實可以使用window函數來實現您想要的功能

#standardSQL
SELECT day,
  (SELECT HLL_COUNT.MERGE(sketch) FROM UNNEST(rolling_sketch_arr) sketch)  rolling_sketch
FROM (
  SELECT day, 
    ARRAY_AGG(ids_sketch) OVER(ORDER BY UNIX_DATE(day) RANGE BETWEEN 89 PRECEDING AND CURRENT ROW) rolling_sketch_arr 
  FROM (
    SELECT day, HLL_COUNT.INIT(id) ids_sketch
    FROM `project.dataset.table`
    GROUP BY day
  )
)

您可以使用[全部]偽數據來測試，玩游戲，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 id, DATE '2019-01-01' day UNION ALL
  SELECT 2, '2019-01-01' UNION ALL
  SELECT 3, '2019-01-01' UNION ALL
  SELECT 1, '2019-01-02' UNION ALL
  SELECT 4, '2019-01-02' UNION ALL
  SELECT 2, '2019-01-03' UNION ALL
  SELECT 3, '2019-01-03' UNION ALL
  SELECT 4, '2019-01-03' UNION ALL
  SELECT 5, '2019-01-03' UNION ALL
  SELECT 1, '2019-01-04' UNION ALL
  SELECT 4, '2019-01-04' UNION ALL
  SELECT 2, '2019-01-05' UNION ALL
  SELECT 3, '2019-01-05' UNION ALL
  SELECT 5, '2019-01-05' UNION ALL
  SELECT 6, '2019-01-05' 
)
SELECT day,
  (SELECT HLL_COUNT.MERGE(sketch) FROM UNNEST(rolling_sketch_arr) sketch)  rolling_sketch
FROM (
  SELECT day, 
    ARRAY_AGG(ids_sketch) OVER(ORDER BY UNIX_DATE(day) RANGE BETWEEN 2 PRECEDING AND CURRENT ROW) rolling_sketch_arr 
  FROM (
    SELECT day, HLL_COUNT.INIT(id) ids_sketch
    FROM `project.dataset.table`
    GROUP BY day
  )
)
-- ORDER BY day

結果

Row day         rolling_sketch   
1   2019-01-01  3    
2   2019-01-02  4    
3   2019-01-03  5    
4   2019-01-04  5    
5   2019-01-05  6    

Answer 2

結合HLL_COUNT.INIT和HLL_COUNT.MERGE 。 此解決方案使用90天的交叉聯接與GENERATE_ARRAY(1, 90)而不是OVER 。

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , HLL_COUNT.MERGE(sketch) unique_90_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
FROM (
  SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
ORDER BY date_grp