提高“反向”LIKE查询的性能

Question

I have a query that looks similar to this:

SELECT  CustomerId
FROM    Customer cust
WHERE   'SoseJost75G' LIKE cust.ClientCustomerId + '%' -- ClientCustomerId is SoseJost 

The jist of what this does is, I get a value from the customer that is my ClientCustomerId but with an unknown number of extra chars attached to the end.

So in my example, the customer gives me SoseJost75G but my database only has SoseJost (without the 75G on the end.)

My query works works. But it takes over a minute to run. That is because it can't use the index that is on ClientCustomerId.

Does anyone know a way to improve the performance of this kind of query?

Answer 1

You might try something like this:

DECLARE @var VARCHAR(100)='SoseJost75G';

WITH pre_selected AS
(SELECT * FROM Customer WHERE ClientCustomerId  LIKE LEFT(@var,6) + '%')
SELECT * 
FROM pre_selected WHERE @var LIKE ClientCustomerId +'%';

With a LIKE with fix start -search an existing index on ClientCustomerId will be used.

With a CTE you never know exactly, which order of execution will take place, but - in some quick test - the optimizer chose first to reduce the set to a tiny rest and perform the heavy search as second step.

If the order of execution is not the way you expect this, you might insert the result of the first CTE-query into a declared variable (only the column with the ID) and then continue with this tiny table...

Something like this

DECLARE @var VARCHAR(100)='SoseJost75G';

DECLARE @CustIDs TABLE(ClientCustomerID VARCHAR(100));
INSERT INTO @CustIDs(ClientCustomerID)
SELECT ClientCustomerID FROM Customer WHERE ClientCustomerId LIKE LEFT(@var,6) + '%';

--Use this with an IN-clause then
SELECT ClientCustomerId 
FROM @CustIDs WHERE @var LIKE ClientCustomerID +'%'

Answer 2

So, the query to check an actual value is super fast (index seek). So I am going to try out just running a bunch of separate select statements till I find a match.

DECLARE @customerIdSubstring varchar(255) = 'SoseJost75G'
DECLARE @customerIdSubstringLength INT
DECLARE @results TABLE 
(
    CustomerId varchar(255)
)


DECLARE @FoundResults BIT = 0;

WHILE (@FoundResults = 0)
BEGIN 

    INSERT INTO @results (CustomerId)
    SELECT  CustomerId
    FROM    Customer cust
    WHERE   CustomerId = @customerIdSubstring 


    SELECT @FoundResults = CASE 
                               WHEN EXISTS(SELECT * FROM @results) THEN CAST(1 AS BIT)
                               ELSE CAST(0 AS BIT)
                           END

    SET @customerIdSubstringLength = LEN(@customerIdSubstring)

    -- We don't want to match on fewer than 3 chars.  (May not be correct at that point.)
    IF (@customerIdSubstringLength < 3)
        BREAK;

    SET @customerIdSubstring = LEFT(@customerIdSubstring, @customerIdSubstringLength - 1)
END 

SELECT CustomerId
FROM @results

While it is possible that I will run the query many times. Inpractice, it will be 3-6 times per value. I think 3-6 index seeks are better than 1 seek and 1 scan.

This also has the added benefit of returning only the most "LIKE" rows. (Meaning that rows that have SanJos will not return if there are rows that have SanJost .)

Answer 3

If you can specify the minimum ClientCustomerId length, eg it can never be less than four characters, you can limit the results thus:

WHERE ClientCustomerId like left('SoseJost75G', 4) + '%'

Here an index can be used to get the matching records. Your criteria

AND ClientCustomerId <= 'SoseJost75G' and ClientCustomerId

would then have to be looked up only in the records already found.

The complete query:

SELECT CustomerId
FROM Customer cust
WHERE ClientCustomerId like left('SoseJost75G', 4) + '%'
AND ClientCustomerId <= 'SoseJost75G' and ClientCustomerId;

BTW: Your criteria can also be written as

ClientCustomerId = left('SoseJost75G', length(ClientCustomerId))

but I suppose that this isn't faster than your version.

Answer 4

I liked your approach, Vaccano. I just simplified it a bit, in case you are interested:

DECLARE @customerIdSubstring varchar(255) = 'SoseJost75G'
DECLARE @results TABLE 
(
    CustomerId varchar(255)
)

DECLARE @FoundResults BIT = 0
DECLARE @customerIdSubstringLength INT = LEN(@customerIdSubstring)

WHILE (@FoundResults = 0 AND @customerIdSubstringLength >= 3)
BEGIN 
    INSERT INTO @results
    SELECT  CustomerId
    FROM    Customer
    WHERE   CustomerId = @customerIdSubstring

    -- Make @FoundResults = 1 if there's at least one record
    SELECT TOP 1 @FoundResults = 1 FROM @results

    SET @customerIdSubstringLength = @customerIdSubstringLength - 1
    SET @customerIdSubstring = LEFT(@customerIdSubstring, @customerIdSubstringLength)
END

SELECT CustomerId
FROM @results

If you are totally sure that only one ID will match, you can simplify this even further by removing the results table, which would only have one row. I also removed the assignment to @customerIdSubstring in the loop:

DECLARE @customerIdSubstring varchar(255) = 'SoseJost75G'
DECLARE @customerIdFound varchar(255)

DECLARE @customerIdSubstringLength INT = LEN(@customerIdSubstring)

WHILE (@customerIdFound IS NULL AND @customerIdSubstringLength >= 3)
BEGIN 
    SELECT  @customerIdFound = CustomerId
    FROM    Customer
    WHERE   CustomerId = LEFT(@customerIdSubstring, @customerIdSubstringLength)

    SET @customerIdSubstringLength = @customerIdSubstringLength - 1
END

SELECT @customerIdFound

Answer 5

Basically, with your statement there is nothing wrong, because you could write it as a sargable query.

SARG = Search Argument

A sargable query allows the optimizer to use indices, while for not sargable queries the optimizer will have to scan all rows in the table, even indexes are available.

A LIKE with the % at the end is sargable. A LIKE withe the % at the beginning is NOT sargable. Applying a function like LEFT([Column], 4) + '%' in the WHERE clause makes the query not sargable. At least the documentation on SARG says so.

[COLUMN] LIKE 'abc%' -> sargable
[COLUMN] LIKE '%abc' -> not sargable
[COLUMN] LIKE LEFT('ABCDE', 4) -> not sargable

I think you should redesign the process before starting any query. Setup a proper ETL-Porcess for separating the ID and the suffix. Store that data in separate columns and configure the indices as required. Than run your query on the transformed data.

This is all the more the preferred process, because you do not know what data you get.