How to search huge data on multiple column with different datatypes?

Question

I have a DataElement table like this:

 DataElement table 
 ValueAsBoolean : bit nullable
 ValueAsDatetime : datetime2 nullable
 ValueAsDecimal : decimal(10,3) nullable
 ValueAsInt : int nullable
 ValueAsString : nvarchar(100) nullable

The data will look like this:

ValueAsBoolean	ValueAsDateTime	ValueAsDecimal	ValueAsInt	ValueAsString
1	NULL	NULL	NULL	NULL
NULL	2022-09-03 12:30:01.210	NULL	NULL	NULL
NULL	NULL	23.000	NULL	NULL
NULL	NULL	NULL	5	NULL
NULL	NULL	NULL	NULL	MONTARY INC.

I want to search data across all columns that match the input search keyword.

• for ValueAsDatetime can be searched by TIME ZONE and format show on UI
• for ValueAsBoolean can be searched by text "TRUE", "FALSE" or part of "TRUE" or "FALSE"
• for ValueAsString , ValueAsInt , ValueAsDecimal can be search by part of input text

For example given 2022-09-03 12:30:01.210 is Tue, 9 Sep 2022

User input "T".

Output will be rows

• 1 because "T" match with ValueAsBoolean 1 ( T RUE)
• 2 because "T" match with " T ue, 9 Sep 2022"
• 5 because "T" match with "MON T ARY INC."

So, I create query base on EF core by:

1. convert all column into String value
2. format the value
3. check value contains the search keyword or not

The query will look like:

SELECT *
FROM [DataElement]
WHERE 
    (([ValueAsBoolean] IS NOT NULL AND (('T' = N'')
    OR (CHARINDEX('T', LOWER( CASE WHEN ValueAsBoolean = 1 then 'true' else 'false' END)) > 0))) 
    OR ([ValueAsDateTime] IS NOT NULL AND (('T' = N'') 
    OR (CHARINDEX('T', LOWER( FORMAT(CONVERT(DATETIME, SWITCHOFFSET( [ValueAsDateTime], DATEPART(TZOFFSET, ValueAsDateTime AT TIME ZONE 'UTC' ))), 'ddd, d MMM yyyy H:mm:ss' ))) > 0)))) 
    OR ([ValueAsDecimal] IS NOT NULL AND (('T' = N'')
    OR (CHARINDEX('T', LOWER(CONVERT(VARCHAR(100), [ValueAsDecimal]))) > 0)))) 
    OR ([ValueAsInt] IS NOT NULL AND (('T' = N'')
    OR (CHARINDEX('T', LOWER(CONVERT(VARCHAR(11), [ValueAsInt]))) > 0)))) 
    OR ([ValueAsString] IS NOT NULL AND (('T' = N'')
    OR (CHARINDEX('T', LOWER([ValueAsString])) > 0)))

The problem is that when the data is huge (more than 500000 records) it takes a long time to process (more than 30 seconds) and exceeds the query timeout limit.

My question is how to search huge data on multiple columns like this?

The one option I think is to create new column as NVARCHAR(100), convert and save data on multiple COLUMN into this column, set the index to the column and search only that column, but I cannot apply search format for datetime.

Or is there another solution for this?

Answer 1

Try following code. With this code, you can search in all the fields of all the database tables. With a little change, you can add the table name as a filter and search only in the desired table:

DECLARE @SearchStr nvarchar(100)
SET @SearchStr = 'Put Search String Here'
CREATE TABLE #Results (ColumnName nvarchar(370), ColumnValue nvarchar(3630))
SET NOCOUNT ON
DECLARE @TableName nvarchar(256), @ColumnName nvarchar(128), @SearchStr2 nvarchar(110)
SET  @TableName = ''
SET @SearchStr2 = QUOTENAME('%' + @SearchStr + '%','''')
WHILE @TableName IS NOT NULL
BEGIN
    SET @ColumnName = ''
    SET @TableName =
    (
        SELECT MIN(QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME))
        FROM     INFORMATION_SCHEMA.TABLES
        WHERE         TABLE_TYPE = 'BASE TABLE'
            AND    QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME) > @TableName
            AND    OBJECTPROPERTY(
                    OBJECT_ID(
                        QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME)
                         ), 'IsMSShipped'
                           ) = 0
    )
    WHILE (@TableName IS NOT NULL) AND (@ColumnName IS NOT NULL)
    BEGIN
        SET @ColumnName =
        (
            SELECT MIN(QUOTENAME(COLUMN_NAME))
            FROM     INFORMATION_SCHEMA.COLUMNS
            WHERE         TABLE_SCHEMA    = PARSENAME(@TableName, 2)
                AND    TABLE_NAME    = PARSENAME(@TableName, 1)
                AND    DATA_TYPE IN ('char', 'varchar', 'nchar', 'nvarchar', 'int', 'decimal')
                AND    QUOTENAME(COLUMN_NAME) > @ColumnName
        )
        IF @ColumnName IS NOT NULL
        BEGIN
            INSERT INTO #Results
            EXEC
            (
                'SELECT ''' + @TableName + '.' + @ColumnName + ''', LEFT(' + @ColumnName + ', 3630) FROM ' + @TableName + ' (NOLOCK) ' +
                ' WHERE ' + @ColumnName + ' LIKE ' + @SearchStr2
            )
        END
    END  
END
SELECT ColumnName, ColumnValue FROM #Results
DROP TABLE #Results