使用选择概率优化 T-SQL 查询

Question

I have a table with entries from which two entries are supposed to be selected. The probability of some entries to be selected should be higher than the probability of others.

Currently I solve this with UNION ALL accordingly I select once all entries and then again the entries which should have a higher probability. From this merged table I select then after the call of ORDER BY NEWID() for mixing with TOP 2 two entries.

SELECT TOP 2 EMail 
FROM (
   SELECT EMail 
   FROM dbo.Benutzer 
   UNION ALL 
   SELECT EMail 
   FROM dbo.Benutzer 
   WHERE param1 = 1 
   UNION ALL 
   SELECT EMail 
   FROM dbo.Benutzer 
   WHERE param2 = 1
) AS EMail 
ORDER BY NEWID();

Example table:

EMail           param1      param2
______________|_________|___________
Test@test.com |0        | 0             -> probability is 1 (normal)
Test1@test.com|1        | 0             -> probability is 2 (higher than 1)
Test2@test.com|1        | 0             -> probability is 2 (higher than 1)
Test3@test.com|1        | 1             -> probability is 3 (higher than 1 and 2)
Test4@test.com|1        | 0             -> probability is 2 (higher than 1)
Test5@test.com|1        | 0             -> probability is 2 (higher than 1)

so if I make a select from this table and shuffle it before that, that with each new select different data comes out, the probability should be based on this table. So how likely it is that this entry comes up.

Always two different users should come out. eg Test3 and Test4. However, also Test unf Test3 so it should be only a probability.

However, this query is not performat. How can this problem be solved in a performant way?

Answer 1

Updated answer:

It seems that you want to randomly choose one row per e-mail with higher probabilty. In this situation you need to number the rows for each different e-mail:

SELECT *
INTO Benutzer
FROM (VALUES
   ('Test@test.com',  0, 0),             
   ('Test1@test.com', 1, 0),             
   ('Test2@test.com', 1, 0),             
   ('Test3@test.com', 1, 1),             
   ('Test4@test.com', 1, 0),             
   ('Test5@test.com', 1, 0)              
) v (EMail, param1, param2)

SELECT TOP 2 Email
FROM (
   SELECT b.*, ROW_NUMBER() OVER (PARTITION BY b.Email ORDER BY a.probability DESC) AS rn
   FROM Benutzer b
   CROSS APPLY (VALUES
      (1), -- normal probability
      (
      1 + 
      CASE WHEN param1 = 1 THEN 1 ELSE 0 END + 
      CASE WHEN param2 = 1 THEN 1 ELSE 0 END
      )    -- calculated probability
   ) a (probability)
) t
WHERE t.rn = 1
ORDER BY NEWID()

Finally, you may try to number the rows randomly:

ROW_NUMBER() OVER (PARTITION BY b.Email ORDER BY NEWID()) AS rn

Original answer:

If I understand the question correctly, you simply need to calculate the probability for each row based on desired conditions:

SELECT TOP 2 EMail
FROM (VALUES
   ('Test@test.com',  0, 0),             
   ('Test1@test.com', 1, 0),             
   ('Test2@test.com', 1, 0),             
   ('Test3@test.com', 1, 1),             
   ('Test4@test.com', 1, 0),             
   ('Test5@test.com', 1, 0)              
) Benutzer (EMail, param1, param2)
ORDER BY 
   (
   1 +
   CASE WHEN param1 = 1 THEN 1 ELSE 0 END +
   CASE WHEN param2 = 1 THEN 1 ELSE 0 END
   ) DESC,
   NEWID()

An important note (based on the @GordonLinoff's comment) - if NEWID() is the first order expression, the second order by expression is ignored, so I placed NEWID() as second order expression (and this was the first version of the answer). But, in this case, the rows are not returned randomly.

Answer 2

You can try using a tally table and ROW_NUMBER() functions. The Tally table is used to create the correct number of duplicate rows for each row based upon probability as given. The ROW_NUMBER() function creates to row numbers one based upon a sort order of NewId() - this is RN, the other is based upon the ordering within an email address - this is RNK.

We want to select 2 rows ordered by RN, but use RNK = 1 to ensure we don't receive duplicates.

SQL Fiddle

MS SQL Server 2017 Schema Setup :

CREATE TABLE Benutzer
(
    Email VARCHAR(30),
    Param1 Int,
    Param2 Int
);

INSERT INTO Benutzer
VALUES
       ('Test@test.com',  0, 0),       -- probability 1      
       ('Test1@test.com', 1, 0),       -- probability 2      
       ('Test2@test.com', 1, 0),       -- probability 2      
       ('Test3@test.com', 1, 1),       -- probability 3      
       ('Test4@test.com', 1, 0),       -- probability 2      
       ('Test5@test.com', 1, 0)        -- probability 2      
    

Query 1 :

WITH Tally As
(
    SELECT *
    FROM (VALUES
        (1),             
        (2),             
        (3)          
        ) Tally (Num)
),
Results AS
(
    SELECT Email, ROW_NUMBER() OVER (ORDER By NewId()) AS RN,
           ROW_NUMBER() OVER (PArtition BY Email ORDER BY PAram1) As Rnk
    FROM Benutzer
    INNER JOIN Tally
        ON TAlly.Num <= 1 + Param1 + Param2
)
SELECT TOP 2 Email
FROM Results
WHERE Results.Rnk =1
ORDER BY RN

Results :

|          Email |
|----------------|
| Test2@test.com |
| Test5@test.com |

Answer 3

Taking the probability into account. Using a recursion because the next step depends on the result of the previous one, ignore new random index if it points to the same range.

   with weighted as (
      -- weight = f (param1, param2) , using 1 + param1 + param2 here
      select Email, param1, param2, count(*) * (1 + param1 + param2) n
      from Benutzer
      group by Email, param1, param2
   ), ranges as (
     --  a range width = weight
     select Email, sum(n) over(order by Email)- n + 1 n1, sum(n) over(order by Email) n2, sum(n) over() sn
     from weighted
   ), h as (
     select 1 level, t2.*, ABS(CHECKSUM(NewId())) r
         -- , @rnd  % t2.sn + 1  k
     from ranges t2 
     where (@rnd  % t2.sn + 1) between t2.n1 and t2.n2 

     union all
     -- try next random index till the next row is a new one
     select case when t2.Email = h.Email then level else level + 1 end
         , t2.* , ABS(CHECKSUM(NewId()))
         -- ,  r % t2.sn + 1
     from ranges t2 
     join h on r % t2.sn + 1  between t2.n1 and t2.n2 
     where level <=1
   )
   -- take one row from every level
   select top(1) with ties Email, @cnt
   from h
   order by row_number() over(partition by level order by newid());

See test db<>fiddle

500 runs statistics:

Email   n   p
Test@test.com   68  6.800000000000
Test1@test.com  179 17.900000000000
Test2@test.com  159 15.900000000000
Test3@test.com  246 24.600000000000
Test4@test.com  180 18.000000000000
Test5@test.com  168 16.800000000000

Not ideal but close.

Answer 4

What's the main problem?

When using ORDER BY NEWID() approach the ORDER BY clause will cause all records in the result set be sorted which can be a very expensive operation (can use a lot of disk I/O) when the base table has a big number of records.

My suggested solution:

If we add an ID column to the base table like this:

CREATE TABLE #T
(
  ID int IDENTITY(1, 3) PRIMARY KEY,
  Email varchar(100),
  Param1 int,
  Param2 int
);

Then we can use this query to get the desired result (without duplicated values):

WITH RndRange AS
(
    SELECT MIN(ID) RangeStart, MAX(ID) + 2 RangeEnd FROM #T
)
,RndSelector AS
(
    SELECT NULL AS HitID, 0 AS HitCounter, CAST('' AS varchar(100)) AS HitIDList
    UNION ALL
    SELECT 
        NewHitID,
        HitCounter + IIF(HitID IS NULL, 0, 1), 
        CAST(HitIDList + IIF(NewHitID IS NULL, '', CONCAT(NewHitID, ',')) AS varchar(100))
    FROM 
        RndSelector 
    CROSS APPLY
    (
        SELECT ABS(CHECKSUM(NEWID()) % (RangeEnd - RangeStart + 1)) + RangeStart AS RndNum FROM RndRange
    ) C
    OUTER APPLY
    (
        SELECT 
            ID AS NewHitID 
        FROM 
            #T 
        WHERE 
            (ID BETWEEN RndNum - 2 AND RndNum) --Added just for performance
            AND
            (RndNum BETWEEN ID AND ID + Param1 + Param2) 
            AND 
            ID NOT IN (SELECT Value FROM string_split(HitIDList, ','))
    ) O
    WHERE 
        HitCounter + IIF(HitID IS NULL, 0, 1) < 2
)
SELECT * FROM #T WHERE ID IN (SELECT HitID FROM RndSelector)

How does it work?

Suppose we have these records in the table:

ID  Email   Param1  Param2
1   User1   1       1
4   User2   1       0
7   User3   0       0

Each round the recursive is executed one random int number between 1 and 9 (min ID and max ID + 2) is generated. When the random number is between 1 and 3 then the ID of User1 is selected (hit). If random number is 4 or 5 then User2 is selected and when random number is 6 then nothing is selected so go to the next round. The execution of recursive will continue until 2 different IDs are selected (HitCounter = 2).

And with the help of HitIDList, repeated IDs will not be selected again.

As you see there is no table sort required by using this solution. We just generate a few random numbers and find their related records in which dbms will use index scans to find records. So I expect considerable performance improvement because of reduced I/O operations specially for large tables.

Answer Updated:

I added a condition ID BETWEEN RndNum - 2 AND RndNum to the query and now it is blazing fast. Speed test results on a table with 1000000 records:

Applied Query	Test1	Test2	Test3
ORDER BY NEWID()	477ms	400ms	510ms
Random generator	2ms	3ms	2ms