有效的最近邻搜索稀疏矩阵

Question

I have a large corpus of data (text) that I have converted to a sparse term-document matrix (I am using scipy.sparse.csr.csr_matrix to store sparse matrix). I want to find, for every document, top n nearest neighbour matches. I was hoping that NearestNeighbor routine in Python scikit-learn library ( sklearn.neighbors.NearestNeighbor to be precise) would solve my problem, but efficient algorithms that use space partitioning data structures such as KD trees or Ball trees do not work with sparse matrices. Only brute-force algorithm works with sparse matrices (which is infeasible in my case as I am dealing with large corpus).

Is there any efficient implementation of nearest neighbour search for sparse matrices (in Python or in any other language)?

Thanks.

Answer 1

Late answer: Have a look at Locality-Sensitive-Hashing

Support in scikit-learn has been proposed here and here .

Answer 2

您可以尝试使用TruncatedSVD将高维稀疏数据转换为低维密集数据，然后执行球树。