[英]How to search huge data on multiple column with different datatypes?
I have a DataElement
table like this:我有一个这样的
DataElement
表:
DataElement table
ValueAsBoolean : bit nullable
ValueAsDatetime : datetime2 nullable
ValueAsDecimal : decimal(10,3) nullable
ValueAsInt : int nullable
ValueAsString : nvarchar(100) nullable
The data will look like this:数据将如下所示:
ValueAsBoolean ![]() |
ValueAsDateTime![]() |
ValueAsDecimal![]() |
ValueAsInt ![]() |
ValueAsString![]() |
---|---|---|---|---|
1 ![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
2022-09-03 12:30:01.210 ![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
23.000 ![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
5 ![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
NULL![]() |
MONTARY INC.![]() |
I want to search data across all columns that match the input search keyword.我想在与输入搜索关键字匹配的所有列中搜索数据。
ValueAsDatetime
can be searched by TIME ZONE and format show on UIValueAsDatetime
并在 UI 上显示格式ValueAsBoolean
can be searched by text "TRUE", "FALSE" or part of "TRUE" or "FALSE"ValueAsBoolean
可以通过文本“TRUE”、“FALSE”或“TRUE”或“FALSE”的一部分进行搜索ValueAsString
, ValueAsInt
, ValueAsDecimal
can be search by part of input textValueAsString
, ValueAsInt
, ValueAsDecimal
可以按输入文本的一部分进行搜索For example given 2022-09-03 12:30:01.210 is Tue, 9 Sep 2022例如,给定 2022-09-03 12:30:01.210 是 2022 年 9 月 9 日星期二
User input "T".用户输入“T”。
Output will be rows输出将是行
So, I create query base on EF core by:因此,我通过以下方式创建基于 EF 核心的查询:
The query will look like:查询将如下所示:
SELECT *
FROM [DataElement]
WHERE
(([ValueAsBoolean] IS NOT NULL AND (('T' = N'')
OR (CHARINDEX('T', LOWER( CASE WHEN ValueAsBoolean = 1 then 'true' else 'false' END)) > 0)))
OR ([ValueAsDateTime] IS NOT NULL AND (('T' = N'')
OR (CHARINDEX('T', LOWER( FORMAT(CONVERT(DATETIME, SWITCHOFFSET( [ValueAsDateTime], DATEPART(TZOFFSET, ValueAsDateTime AT TIME ZONE 'UTC' ))), 'ddd, d MMM yyyy H:mm:ss' ))) > 0))))
OR ([ValueAsDecimal] IS NOT NULL AND (('T' = N'')
OR (CHARINDEX('T', LOWER(CONVERT(VARCHAR(100), [ValueAsDecimal]))) > 0))))
OR ([ValueAsInt] IS NOT NULL AND (('T' = N'')
OR (CHARINDEX('T', LOWER(CONVERT(VARCHAR(11), [ValueAsInt]))) > 0))))
OR ([ValueAsString] IS NOT NULL AND (('T' = N'')
OR (CHARINDEX('T', LOWER([ValueAsString])) > 0)))
The problem is that when the data is huge (more than 500000 records) it takes a long time to process (more than 30 seconds) and exceeds the query timeout limit.问题是当数据很大时(超过 500000 条记录)需要很长时间来处理(超过 30 秒)并且超过查询超时限制。
My question is how to search huge data on multiple columns like this?我的问题是如何在这样的多个列上搜索大量数据?
The one option I think is to create new column as NVARCHAR(100), convert and save data on multiple COLUMN into this column, set the index to the column and search only that column, but I cannot apply search format for datetime.我认为的一个选择是将新列创建为 NVARCHAR(100),将多个 COLUMN 上的数据转换并保存到该列中,将索引设置为该列并仅搜索该列,但我不能为日期时间应用搜索格式。
Or is there another solution for this?或者有其他解决方案吗?
Try following code.试试下面的代码。 With this code, you can search in all the fields of all the database tables.
使用此代码,您可以在所有数据库表的所有字段中进行搜索。 With a little change, you can add the table name as a filter and search only in the desired table:
稍作改动,您可以将表名添加为过滤器并仅在所需表中搜索:
DECLARE @SearchStr nvarchar(100)
SET @SearchStr = 'Put Search String Here'
CREATE TABLE #Results (ColumnName nvarchar(370), ColumnValue nvarchar(3630))
SET NOCOUNT ON
DECLARE @TableName nvarchar(256), @ColumnName nvarchar(128), @SearchStr2 nvarchar(110)
SET @TableName = ''
SET @SearchStr2 = QUOTENAME('%' + @SearchStr + '%','''')
WHILE @TableName IS NOT NULL
BEGIN
SET @ColumnName = ''
SET @TableName =
(
SELECT MIN(QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME))
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
AND QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME) > @TableName
AND OBJECTPROPERTY(
OBJECT_ID(
QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME)
), 'IsMSShipped'
) = 0
)
WHILE (@TableName IS NOT NULL) AND (@ColumnName IS NOT NULL)
BEGIN
SET @ColumnName =
(
SELECT MIN(QUOTENAME(COLUMN_NAME))
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = PARSENAME(@TableName, 2)
AND TABLE_NAME = PARSENAME(@TableName, 1)
AND DATA_TYPE IN ('char', 'varchar', 'nchar', 'nvarchar', 'int', 'decimal')
AND QUOTENAME(COLUMN_NAME) > @ColumnName
)
IF @ColumnName IS NOT NULL
BEGIN
INSERT INTO #Results
EXEC
(
'SELECT ''' + @TableName + '.' + @ColumnName + ''', LEFT(' + @ColumnName + ', 3630) FROM ' + @TableName + ' (NOLOCK) ' +
' WHERE ' + @ColumnName + ' LIKE ' + @SearchStr2
)
END
END
END
SELECT ColumnName, ColumnValue FROM #Results
DROP TABLE #Results
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.