在大数据集中查找最接近的向量C＃

Question

I have a big (several million rows) dataset of vectors (ie List < double[] >) and I need to find the closest 1000 vectors to the given vector.

The obvious solution is to calculate the distances for all of them and then sort the array, but I'm not sure if it is the right way to do considering the size of the output array.

Maybe I should routinely remove the farthest vectors in the process of calculating the distances, thus it will be a small set of closest vectors all the time instead of a huge array in the end.

On the other hand, it looks like I still can handle arrays this size without memory overflow error in 64 bit.

What will be the less costly way of solving this problem?

If the right way is to have small set while calculating - what will be the right way of doing that?

Answer 1

如果将数据集存储在数据库中，则大多数现代DBMS支持地理编码和按距离搜索。