在Win 7上具有SQL Server 2008 R2唯一索引列的1000万条记录表中快速搜索

Question

I need to do a fast search in a column with floating point numbers in a table of SQL server 2008 R2 on Win 7.

the table has 10 million records.

eg

  Id    value
  532   937598.32421
  873   501223.3452
  741   9797327.231

ID is primary key, I need o do a search on "value" column for a given value such that I can find the 5 closest points to the given point in the table.

The closeness is defined as the absolute value of the difference between the given value and column value.

The smaller value, the closer.

I would like to use binary search.

I want to set an unique index on the value column.

But, I am not sure whether the table will be sorted every time when I search the given value in the column ?

Or, it only sorts the table one time because I have set the value column as unique index ?

Are there better ways to do this search ?

A sorting will have to be done whenever I do a search ? I need to do a lot of times of search in the table. I know the sorting time is O(n lg n). Using index can really have done the sort for me ? or the index is associated with a sorted tree to hold the column values ?

When an index is set up, the values have been sorted ? I do not need to sort it every time when I do a search ?

Any help would be appreciated.

thanks

Answer 1

Sorry for my initial response, no, I would not even create an index, it won't be able to use it because you're searching not on a given value but the difference between that given value and the value column on the table. You could create a function based index, but you would have to specify the # you're searching on, which is not constant.

Given that, I would look at getting enough RAM to swallow the whole table. Ie. if the table is 10gb, try to get 10gb ram allocated for caching. And if possible do it on a machine w/ an SSD, or get an SSD.

The sql itself is not complicated, it's really just an issue of performance.

select top 5 id, abs(99 - val) as diff
from tbl
order by 2

If you don't mind some trial and error, you could create an index on the value column, and then search as follows -

select top 5 id, abs(99 - val) as diff
from tbl
where val between 99-30 and 99+30
order by 2

The above query WOULD utilize the index on the value column, because it is searching on a range of values in the value column, not the differences between the values in that column and X (2 very different things)

However, there is no guarantee it would return 5 rows, it would only return 5 rows if there actually existed 5 rows within 30 of 99 (69 to 129). If it returned 2, 3, etc. but not 5, you would have to run the query again and expand the range, and keep doing so until you determine your top 5. However, these queries should run quite a bit faster than having no index and firing against the table blind. So you could give it a shot. The index may take a while to create though, so you might want to do that part overnight.

Answer 2

You mention sql server and binary search. SQL server does not work that way, but sql server (or other database) is a good solution for this problem.

Just to concrete, I will assume

create table mytable
(
  id int not null
, value float not null
  constraint mytable_pk primary key(id)
)

And you need an index on the value field.

Now get ten rows 5 above and 5 below the search value with these 2 selects

  SELECT TOP 5 id, value, abs(id-value) as diff
      FROM mytable
      WHERE value >= @searchval
      ORDER BY val asc) as bigger

  -- and 

  SELECT TOP 5 id, value, abs(id-value) as diff
      FROM mytable
      WHERE value < @searchval
      ORDER BY val desc) as smaller

To combine the 2 unions into 1 result set you need

SELECT *
  FROM (SELECT TOP 5 id, value, abs(id-value) as diff
          FROM mytable
         WHERE value >= @searchval
      ORDER BY val asc) as bigger
UNION ALL
  FROM (SELECT TOP 5 id, value, abs(id-value) as diff
          FROM mytable
         WHERE value < @searchval
      ORDER BY val desc) as smaller

But since you only want the smallest 5 differences, wrap with one more layer as

SELECT TOP 5 * FROM
(
SELECT *
  FROM (SELECT TOP 5 id, value, abs(id-value) as diff
          FROM mytable
         WHERE value >= @searchval
      ORDER BY val asc) as bigger
UNION ALL
  FROM (SELECT TOP 5 id, value, abs(id-value) as diff
          FROM mytable
         WHERE value < @searchval
      ORDER BY val desc) as smaller
)
ORDER BY DIFF ASC

I Have not tested any of this

Answer 3

Creating the table's clustered index upon [value] will cause [value]'s values to be stored on disk in sorted order. The table's primary key (perhaps already defined on [Id]) might already be defined as the table's clustered index. There can only be one clustered index on a table. If a primary key on [Id] is already clustered, the primary key will need to be dropped, the clustered index on [value] will need to be created, and then the primary key on [Id] can be recreated (as a nonclustered primary key). A clustered index upon [value] should improve performance of this specific statement, but you must ultimately test all variety of T-SQL that will reference this table before making the final choice about this table's most useful clustered index column(s).

Because the FLOAT data type is imprecise (subject to your system's FPU and its floating point rounding and truncation errors, while still in accordance with IEEE 754's specifications), it can be a fatal mistake to assume every [value] will be unique, event when the decimal number (being inserted into FLOAT) appears (in decimal) to be unique. Irrational numbers must always be truncated and rounded. In decimal, PI is an example of an irrational value, which can be truncated and rounded to an imprecise value of 3.142. Similarly, the decimal number 0.1 is an irrational number in binary, which means FLOAT will not store decimal 0.1 as a precise binary value.... You might want to consider whether the domain of acceptable values offered by the NUMERIC data type can accommodate [value] (thus gaining more precise answers when compared to a use of FLOAT).

While a NUMERIC data type might require more storage space than FLOAT, the performance of a given query is often controlled by the levels of the (perhaps clustered) index's B-Tree (assuming an index seek can be harnessed by the query, which for your specific need is a safe assumption). A NUMERIC data type with a precision greater than 28 will require 17 bytes of storage per value. The payload of SQL Server's 8KB page is approximately 8000 bytes. Such a NUMERIC data type will thus store approximately 470 values per page. A B-Tree will consume 2^(index_level_pages-1) * 470 rows/page to store the 10,000,000 rows. Dividing both sides by 470 rows/page: 2^(index_level_pages-1) = 10,000,000/470 pages. Simplifying: log(base2)10,000,000/470 = (index_level_pages-1). Solving: ~16 = index_level_pages (albeit this is back of napkin math, I think it close enough). Thus searching for a specific value in a 10,000,000 row table will require ~16*8KB = ~128 KB of reads. If a clustered index is created, the leaf level of a clustered index will contain the other NUMERIC values that are "close" to the one being sought. Since that leaf level page (and the 15 other index pages) are now cached in SQL Server's buffer pool and are "hot", the next search (for values that are "close" to the value being sought) is likely to be constrained by memory access speeds (as opposed to disk access speeds). This is why a clustered index can enhance performance for your desired statement.

If the [value]'s values are not unique (perhaps due to floating point truncation and rounding errors), and if [value] has been defined as the table's clustered index, SQL Server will (under the covers) add a 4-byte "uniqueifier" to each value. A uniqueifier adds overhead (per above math, it is less overhead than might be thought, when a index can be harnessed). That overhead is another (albeit less important) reason to test. If values can instead be stored as NUMERIC and if a use of NUMERIC would more precisely ensure persisted decimal values are indeed unique (just the way they look, in decimal), that 4 byte overhead can be eliminated by also declaring the clustered index as being unique (assuming value uniqueness is a business need). Using similar math, I am certain you will discover the index levels for a FLOAT data type are not all that different from NUMERIC.... An index B-Tree's exponential behavior is "the great leveler" :). Choosing FLOAT because it has smaller storage space than NUMERIC may not be as useful as can initially be thought (even when greatly more storage space for the table, as a whole, is needed).

You should also consider/test whether a Columnstore index would enhance performance and suit your business needs.

Answer 4

This is a common request coming from my clients.

It's better if you transform your float column into two integer columns (one for each part of the floating point number), and put the appropriate index on them for fast searching. For example: 12345.678 will become two columns 12345 and 678.