SELECT查询SQL服务器的效率

Question

My friend and I tried to build an advanced model for soccer pools betting. Due to limitations in Excel we found SQL to be a better choice going forward. We've managed to achieve what we were aiming for but the process takes around 20-30 minutes each time which I think it because of the inefficiency in the process. I'll try to explain what we are doing and hopefully you smart guys can point me in the right direction to do a more efficient approach.

So let's start to show you how the SQL-database looks like. We got one main table, Rows , which contains all possible combinations of game outcomes for 13 soccer matches:

CREATE TABLE [dbo].[Rows](
[RowID] [int] NULL,
[Match_1] [int] NULL,
[Match_2] [int] NULL,
[Match_3] [int] NULL,
[Match_4] [int] NULL,
[Match_5] [int] NULL,
[Match_6] [int] NULL,
[Match_7] [int] NULL,
[Match_8] [int] NULL,
[Match_9] [int] NULL,
[Match_10] [int] NULL,
[Match_11] [int] NULL,
[Match_12] [int] NULL,
[Match_13] [int] NULL
);

INSERT INTO Rows
    (RowID, Match_1, Match_2, Match_3, Match_4, Match_5, Match_6, Match_7, Match_8, Match_9, Match_10, Match_11, Match_12, Match_13) 
VALUES 
(1,1,1,1,1,1,1,1,1,1,1,1,1,1),
(2,1,1,1,1,1,1,1,1,1,1,1,1,3),
(3,1,1,1,1,1,1,1,1,1,1,1,1,2),
(4,1,1,1,1,1,1,1,1,1,1,1,3,1),
(5,1,1,1,1,1,1,1,1,1,1,1,3,3),
(6,1,1,1,1,1,1,1,1,1,1,1,3,2),
(7,1,1,1,1,1,1,1,1,1,1,1,2,1),
(8,1,1,1,1,1,1,1,1,1,1,1,2,3),
(9,1,1,1,1,1,1,1,1,1,1,1,2,2),
(10,1,1,1,1,1,1,1,1,1,1,3,1,1);

So this amounts to around 1,6 million rows. The values stand for 1=Home victory, 3=Draw, 2=Away victory.

Now we need to select which ones are likely be the outcome for the current week. We use Excel to have our data and track which conditions to be used. Excel populates SQL-queries with the numbers adjusted for each gameweek so we just copy and paste into SQL Server Management Studio. We have around 1300 conditions which we test and apply approximately 600-700 every week. To be able to add the data for these 13 specific games we use computed columns. So we add around 700 computed columns to the table. Example:

ALTER TABLE dbo.Rows ADD Group1 AS ((CASE WHEN [Match_1] = 1 THEN 0 WHEN [Match_1] = 3 THEN 0 WHEN [Match_1] = 2 THEN 0 END) + (CASE WHEN [Match_2] = 1 THEN 1 WHEN [Match_2] = 3 THEN 0 WHEN [Match_2] = 2 THEN 0 END) + (CASE WHEN [Match_3] = 1 THEN 0 WHEN [Match_3] = 3 THEN 0 WHEN [Match_3] = 2 THEN 0 END) + (CASE WHEN [Match_4] = 1 THEN 0 WHEN [Match_4] = 3 THEN 0 WHEN [Match_4] = 2 THEN 0 END) + (CASE WHEN [Match_5] = 1 THEN 0 WHEN [Match_5] = 3 THEN 0 WHEN [Match_5] = 2 THEN 0 END) + (CASE WHEN [Match_6] = 1 THEN 0 WHEN [Match_6] = 3 THEN 0 WHEN [Match_6] = 2 THEN 0 END) + (CASE WHEN [Match_7] = 1 THEN 0 WHEN [Match_7] = 3 THEN 0 WHEN [Match_7] = 2 THEN 0 END) + (CASE WHEN [Match_8] = 1 THEN 0 WHEN [Match_8] = 3 THEN 0 WHEN [Match_8] = 2 THEN 0 END) + (CASE WHEN [Match_9] = 1 THEN 0 WHEN [Match_9] = 3 THEN 0 WHEN [Match_9] = 2 THEN 0 END) + (CASE WHEN [Match_10] = 1 THEN 0 WHEN [Match_10] = 3 THEN 0 WHEN [Match_10] = 2 THEN 0 END) + (CASE WHEN [Match_11] = 1 THEN 0 WHEN [Match_11] = 3 THEN 0 WHEN [Match_11] = 2 THEN 0 END) + (CASE WHEN [Match_12] = 1 THEN 0 WHEN [Match_12] = 3 THEN 0 WHEN [Match_12] = 2 THEN 0 END) + (CASE WHEN [Match_13] = 1 THEN 0 WHEN [Match_13] = 3 THEN 0 WHEN [Match_13] = 2 THEN 0 END));

So basically what this does is to, with help of the CASE-expression, calculate how many of the desired outcomes each row has. For Group1 we only want exactly one game which you will see in the SELECT query below.

The last step (which is the one which takes the majority of the time) is to SELECT all rows which fulfils all our requirements. It uses as previously said approx 700 different conditions so we have to split it into several queries using WITH -clauses.

WITH Step1 AS (
SELECT [Rows].[RowID],[Rows].[Match_1],[Rows].[Match_2],[Rows].[Match_3],[Rows].[Match_4],[Rows].[Match_5],[Rows].[Match_6],[Rows].[Match_7],[Rows].[Match_8],[Rows].[Match_9],[Rows].[Match_10],[Rows].[Match_11],[Rows].[Match_12],[Rows].[Match_13]
FROM Rows
WHERE [Rows].[Group1] >= 1 AND [Rows].[Group1] <= 1
)
SELECT [Step1].[RowID],[Step1].[Match_1],[Step1].[Match_2],[Step1].[Match_3],[Step1].[Match_4],[Step1].[Match_5],[Step1].[Match_6],[Step1].[Match_7],[Step1].[Match_8],[Step1].[Match_9],[Step1].[Match_10],[Step1].[Match_11],[Step1].[Match_12],[Step1].[Match_13]
INTO FinalRows
FROM Step1
;

Where should I look to simplify this and gain efficiency? Do you have any suggestions for me going forward? The ideal would be to achieve start to finish on 5-10 minutes max.

Answer 1

TLDR;

Use PERSISTED computed columns, also check the execution plan for hints on where you can optimise your query or data structures.

---

There are far more efficient ways to do this, the first change you should consider is storing ALL the results into the SQL database, it looks like you have only modelled the outcomes over the entire pool for a single round.

I and other commenters have question the validity of such a model, especially when the additional meta-data like team identity, current position on the ladder, previous win loss ration for the season or against the same team are not taken into account. By storing the data such that you only record the results of each match additional insights can be gained and used when or if needed in your betting application.

In Australia we simply call this "Tipping", everyone has a go designing something like this at some point, you will find out that simply from the raw results of each round you can write efficient set-based queries to come up with all sorts of derivative models. You might not even go back to excel for data management, and instead only use it for data visualization...

The first inefficiency is that the data is not all in the database, or if it is, it is not stored in an efficient manner. With simple indexes you should be able to get very efficient queries over many millions of records. 1.6 Million rows with only 14 columns is not a very big record set. Its not small, but you should still expect sub-second response on standard hardware for simple queries. 700 columns, thats a lot of data, but if the columns are defined as TINYINT then storage is not a big deal and you should still get very fast results.

But it sounds like your excel sheet is generating a batch of individual queries. There is a cost imposed simply for establishing the connection, parsing the query and returning the results, this is sometimes insignificant for a single query, but if you are issuing many queries then this handshaking and transmission cost can add up. You should try to alter the individual queries into more of a set-based query.

• This again is much easier if ALL of the data you have in Excel is migrated into SQL Server, or at least the raw results for each match of each historic round.

The next obvious efficiency to be gained is with PERSISTED computed columns. If you do not persist the values of the computed column, then for every query, and every evaluation within that query the computed column value needs to be resolved, if your 700 columns are computed, then thats a lot of work for the server to re-calculate the same value.

Follow Specify Computed Columns in a Table for how to create PERSISTED computed columns:

ALTER TABLE dbo.Products ADD RetailValue AS (QtyAvailable * UnitPrice * 1.5) PERSISTED

It might be that this is the first change you can make that has significant impact. This offloads the computation to the INSERT command, and it adds a storage footprint, but you will experience must greater read throughput, one might even suggest at least a 700% increase.

The last change is in your CTE query, you have not demonstrated any need for a CTE or nested query, SQL server will probably optimise that away anyway, do not be fooled into thinking that CTE's are cached results of a sub-query, that is not how they are implemented under the hood. But that is out of the scope of this post so just use this to return you final results:

SELECT [Rows].[RowID],[Rows].[Match_1],[Rows].[Match_2],[Rows].[Match_3],[Rows].[Match_4],[Rows].[Match_5],[Rows].[Match_6],[Rows].[Match_7],[Rows].[Match_8],[Rows].[Match_9],[Rows].[Match_10],[Rows].[Match_11],[Rows].[Match_12],[Rows].[Match_13]
INTO FinalRows
FROM Rows
WHERE [Rows].[Group1] >= 1 AND [Rows].[Group1] <= 1

---

A Note about Efficiency

When you want to troubleshoot query performance, the first place to check is the Actual Execution Plan , you can also review the Expected execution plan, which might give you some pointers, but the Actual will give you results of the real plan that was used to resolve the query. For long running queries you will be able to visually identify the bottle-necks, then you can post back to SO or https://dba.stackexchange.com/ for more specific advice.