Querying data from a combination of columns using Like (SQL Server 2005/8)

Question

I have a table with around 15 columns which can be queried with different combination. The table columns for instance is UserID, LocationID, DepartmentID, CoOrdinate1, CoOrdinate2, CoOrdinate3...CoOrdinate15.

To speed up the retrieval of combination of data we have created a Computed field where we are storing the values of these columns in the format :UserID::LocationID::DepartmentID::CoOrdinate1:...:CoOrdinate15: - a sample value would look like :1:100:20:22:39:94:29:..:9:

While this is fine for retrieving data where the index key matches ( = operator) we are exploring the best method to fetch combinations.

For instance if the user queries for UserID = 1 and CoOrdinate = 15 we plan to build a Like condition '%:1::%::%::%::%::%::%::%::%::15:%'.

SQL Server is doing an Index Scan to retrieve the data. From a performance point of view - is there a better way to approach this problem.

Answer 1

From the information supplied, it looks as though there are up to 15 numerical coordinates for each record on the existing table.

Which means that the existing table is not properly normalised.

I strongly recommend restructuring the existing table to be more like:

UserID
LocationID
DepartmentID
CoOrdinate Number
CoOrdinate Value

(Alternatively, retain the existing table without the CoOrdinate fields and add a new table with the UserID, LocationID, DepartmentID combination replaced by the key field from the existing table.)

This should allow for much simpler and more efficient querying of data - indexes on numeric fields are much smaller and faster-accessed than on long string fields.

Answer 2

That computed field makes no sense at all. LIKE queries are very slow. You can have multiple indexes on a table, including multiple columns as required. You'll be much better off relying on SQL Server's own indexing than trying to roll your own.

Answer 3

Try simple approach, I'm wrapping it into SP:

CREATE PROC Find
@UserId int,
@LocationId int,
....
@CoOrdinate15 int
WITH RECOMPILE
AS
BEGIN
  SET NOCOUNT ON;

  SELECT [what you need]
  FROM YourTable
  WHERE 
      (@UserId IS NULL OR UserId = @UserId) 
  AND (@LocationId IS NULL OR LocationId = @LocationId)
  ...
  AND (@CoOrdinate15 IS NULL OR CoOrdinate15  = @CoOrdinate15)
END

RECOMPILE causes Sql Server's optimizer to finely adopt each call to SP taking into account NULL-valued parameters and to choose the right index for every call

Answer 4

No. Leading wildcard LIKE searches will always be rubbish.

Do you really need complete exhaustive search flexibility?

If you are looking for '%:1::%::%::%::%::%::%::%::%::15:%' then you should search correctly with WHERE x=1 and y=15 and add appropiate indexes.

Answer 5

let me sum this up:

• when query a table, in your where clause you can have an arbitrary combination of the columns
• you could create an individual index on each column, but only one of them would be used for your query, as normal b-tree indexes cannot be combined
• you could create composite indexes on certain combination of columns but these indexes would only be used by those queries that have a where clause that matches the columns in the composite index; also, having many wide composite indexes would incur big maintenance overhead
• Since you want to filter against arbitrary combination of the columns in your queries, composite indexes are not an option: you cannot create a composite index for all possible combination of columns

In general, the solution to this problem is having bitmap indexes on each columns, because bitmap indexes can be combined. Unfortunately SQL Server does not support bitmap indexes but I've heard it has some similar feature. I suggest you look into that:

http://msdn.microsoft.com/en-us/library/bb522541.aspx (This article discuss the usage of bitmap indexes when joining tables, but dont let that confuse you, they can be useful in your use case as well, when you query a single table.)

Answer 6

If you insist on the computed field you should construct it such a way:

"field1=value1;field2=value2;....;fieldn=valuen"

that is

"UserID=123;LocationID=34;DepartmentID=2;CoOrdinate1=56;..."

You would define a full text index on the computed field

For instance if the user queries for UserID = 1 and CoOrdinate = 15 your where clause would be

WHERE CONTAINS(computed_field, "UserID=1" AND "CoOrdinate=15")

You will have to take care of properly indexing "=" and numbers when you define your index. You should treat "=" and numbers as part of the words, so "UserID=1" would be one word in the index.

Answer 7

建立以where 1=1开头的where子句，并将每个相关部分附加为and xx_field = 'value'或and xx_field like '%value%'

Answer 8

Erland Sommarskog对您的案例进行了很好的分析： T-SQL中的动态搜索条件