[英]Speeding up SQL Server cross apply to get aggregated data
在SQL Server中,我試圖將一個查詢放在一起,該查詢抓取一行並包含該行之前兩小時窗口的聚合數據以及之后一小時窗口的聚合數據。 如何讓這個運行得更快?
行具有毫秒精度的時間戳,並且間隔不均勻。 我在這個表中有超過5000萬行,查詢似乎沒有完成。 許多地方都有索引,但它們似乎沒有幫助。 我也在考慮使用窗口函數,但我不確定它是否可能有一個不均勻分布的行的滑動窗口。 此外,對於未來的一小時窗口,我不確定如何使用SQL窗口完成。
Box是一個字符串,有10個唯一值。 Process是一個字符串,有30個唯一值。 平均duration_ms是200毫秒。 錯誤占數據的比例不到0.1%。 5000萬行描述了多年的數據。
select
c1.start_time,
c1.end_time,
c1.box,
c1.process,
datediff(ms,c1.start_time,c1.end_time) as duration_ms,
datepart(dw,c1.start_time) as day_of_week,
datepart(hour,c1.start_time) as hour_of_day,
c3.*,
c5.*
from metrics_table c1
cross apply
(select
avg(cast(datediff(ms,c2.start_time,c2.end_time) as numeric)) as avg_ms,
count(1) as num_process_total,
count(distinct process) as num_process_unique,
count(distinct box) as num_box_unique
from metrics_table c2
where datediff(minute,c2.start_time,c1.start_time) <= 120
and c1.start_time> c2.start_time
and c2.error_code = 0
) c3
cross apply
(select
avg(case when datediff(ms,c4.start_time,c4.end_time)>1000 then 1.0 else 0.0 end) as percent_over_thresh
from metrics_table c4
where datediff(hour,c1.start_time,c4.start_time) <= 1
and c4.start_time> c1.start_time
and c4.error_code= 0
) c5
where
c1.error_code= 0
編輯
版本:SQL Azure 12.0
以下應該是朝着正確方向邁出的一步......注意:c2.start_time&c4.start_time不再在DATEDIFF函數中包裝,使它們成為SARGable ......
SELECT
c1.start_time,
c1.end_time,
c1.box,
c1.process,
DATEDIFF(ms, c1.start_time, c1.end_time) AS duration_ms,
DATEPART(dw, c1.start_time) AS day_of_week,
DATEPART(HOUR, c1.start_time) AS hour_of_day,
--c3.*,
avg_ms = CASE WHEN
c5.*
FROM
dbo.metrics_table c1
CROSS APPLY (
SELECT
AVG(CAST(DATEDIFF(ms, c2.start_time, c2.end_time) AS NUMERIC)) AS avg_ms,
COUNT(1) AS num_process_total,
COUNT(DISTINCT process) AS num_process_unique,
COUNT(DISTINCT box) AS num_box_unique
FROM
dbo.metrics_table c2
WHERE
--DATEDIFF(minute,c2.start_time,c1.start_time) <= 120
c2.start_time <= DATEADD(MINUTE, -120, c1.start_time)
--and c1.start_time> c2.start_time
AND c2.error_code = 0
) c3
CROSS APPLY (
SELECT
AVG(CASE WHEN DATEDIFF(ms, c4.start_time, c4.end_time) > 1000 THEN 1.0 ELSE 0.0 END
) AS percent_over_thresh
FROM
dbo.metrics_table c4
WHERE
--DATEDIFF(HOUR, c1.start_time, c4.start_time) <= 1
c4.start_time >= DATEADD(HOUR, 1, c1.start_time)
--and c4.start_time> c1.start_time
AND c4.error_code = 0
) c5
WHERE
c1.error_code = 0;
當然,除非有適當的索引,否則進行查詢SARGable沒有任何好處。 以下內容應該適用於所有3個metrics_table引用...(請參閱當前可用的索引,您可能不需要創建新索引)
CREATE NONCLUSTERED INDEX ixf_metricstable_errorcode_starttime ON dbo.metrics_table (
error_code,
start_time
)
INCLUDE (
end_time,
box,
process
)
WHERE
error_code = 0;
我使用Between
並在我的簡單測試台中獲得了良好的性能。 我還使用了columnstore,因為5000卷記錄是DW卷:
CREATE TABLE dbo.metrics_table (
rowId INT IDENTITY,
start_time DATETIME NOT NULL,
end_time DATETIME NOT NULL,
box VARCHAR(10) NOT NULL,
process VARCHAR(10) NOT NULL,
error_code INT NOT NULL
);
-- Add records
;WITH cte AS (
SELECT TOP 3334 ROW_NUMBER() OVER ( ORDER BY ( SELECT 1 ) ) rn
FROM sys.columns c1
CROSS JOIN sys.columns c2
CROSS JOIN sys.columns c3
)
INSERT INTO dbo.metrics_table ( start_time, end_time, box, process, error_code )
SELECT
DATEADD( ms, rn, DATEADD( day, rn % 365, '1 Jan 2017' ) ) AS start_time,
DATEADD( ms, rn % 409, DATEADD( ms, rn, DATEADD( day, rn % 365, '1 Jan 2017' ) ) ) AS end_time,
'box' + CAST( boxes.box AS VARCHAR(10) ) box,
'process' + CAST( boxes.box AS VARCHAR(10) ) process,
ABS( CAST( rn % 3000 AS BIT ) -1 ) error_code
FROM cte c
CROSS JOIN ( SELECT TOP 10 rn FROM cte ) AS boxes(box)
CROSS JOIN ( SELECT TOP 30 rn FROM cte ) AS processes(process);
-- Create normal clustered index to order the data
CREATE CLUSTERED INDEX cci_metrics_table ON dbo.metrics_table ( start_time, end_time, box, process );
--CREATE CLUSTERED INDEX cci_metrics_table ON dbo.metrics_table ( box, process, start_time, end_time );
-- Convert to columnstore
CREATE CLUSTERED COLUMNSTORE INDEX cci_metrics_table ON dbo.metrics_table WITH ( MAXDOP = 1, DROP_EXISTING = ON );
IF OBJECT_ID('tempdb..#tmp1' ) IS NOT NULL DROP TABLE #tmp1
-- two hour window before, 1 hour window after
SELECT
c1.start_time,
c1.end_time,
c1.box,
c1.process,
DATEDIFF( ms, c1.start_time, c1.end_time ) AS duration_ms,
DATEPART( dw, c1.start_time ) AS day_of_week,
DATEPART( hour, c1.start_time ) AS hour_of_day,
c2.xavg,
c2.num_process_total,
c2.num_process_unique,
c2.num_box_unique,
c3.percent_over_thresh
INTO #tmp1
FROM dbo.metrics_table c1
CROSS APPLY
(
SELECT
COUNT(1) AS num_process_total,
AVG( CAST( DATEDIFF( ms, start_time, end_time ) AS NUMERIC ) ) xavg,
COUNT( DISTINCT process ) num_process_unique,
COUNT( DISTINCT box ) num_box_unique
FROM dbo.metrics_table c2
WHERE c2.error_code = 0
AND c2.start_time Between DATEADD( minute, -120, c1.start_time ) And c1.start_time
AND c1.start_time > c2.start_time
) c2
CROSS APPLY
(
SELECT
AVG( CASE WHEN DATEDIFF( ms, c4.start_time, c4.end_time ) > 1000 THEN 1.0 ELSE 0.0 END ) percent_over_thresh
FROM dbo.metrics_table c4
WHERE c4.error_code = 0
AND c4.start_time Between c1.start_time And DATEADD( minute, 60, c1.start_time )
AND c4.start_time > c1.start_time
) c3
WHERE error_code = 0
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.