没有图像的词袋

Question

I am trying to build a bag of words class wich can create a vocabulary and find a nearest word for the given vector in the vocabulary. For example I load a float vector or a matrix and want to find the nearest word.

In openCV I only found the BOWImgDescriptorExtractor, but this works only with images.

Can someone explain to me how I find the nearest word for a given vector in my vocabulary? I read a lot about the FlannBasedMatcher and the BruteforceMatcher, but I have no clue how to convert the vector to a format for my vocabulary.

Thank you for your help

Answer 1

You want to convert text documents into vectors, where each feature corresponds to a word (or an n-gram, which is a series of n words), and the value for each feature is either the count of the word in the document, or its frequency, or better, it's tf-idf .

Once you have the means to convert a document into a vector, than you can measure the distance between any two vectors. These two vectors represent two different documents. In your case, one vector will be representing a document with a single word, and the other will be the text document you're interested in. To avoid the lengths of documents to play a role in the distance measurement, cosine distance is used a lot in text analysis, rather than euclidean distance .

To find the nearest word to a given vector, you can basically do a brute force search by calculating the cosine distance between each word's vector and the query vector. The word that gives you the smallest distance is the winner.

If you need to do this for a lot of vectors with a big vocabulary, there are algorithms to make this search much faster than brute force. They involve building indexes ( spatial data structures ) that allow you to check the distance of a smaller subset to find the winner (you get to automatically eliminate a whole bunch of words without explicitly measuring the distance). If you're willing to lose a little bit of accuracy to get much much faster in finding the nearest word, there are great algorithms for that as well .

To implement a text document to vector converter, first you need to go over the entire corpus and record every unique word, making a hash table that defines an integer id for each word you see. This is your vocabulary. Let's say there are 50K words. Each of your documents will be represented by a vector that's 50K long. Each vector will be very sparse, you will have 0 for most features (most documents will only have a tiny portion of your entire vocabulary). You will go over each document and calculate the value (either count, frequency, or tf-idf) for each word in the document, and record this value in the vector under the column related to the word in question. This is how you convert text into a vector. A word by itself is the simplest vector, of course: a 1 in the corresponding column and zero everywhere else.

Like stan0 mentioned, word2vec is open source and already does all this, so I'd give it a try. Here is a tutorial to get you started.