类别匹配-正则表达式与全文搜索

Question

I have a fairly large category table with 1500 categories (some singular words others containing multiple) in it and I'm looking for the best way to match new products to these categories by their title.

I've been looking at using regex and looping through the product description for key words but this wouldn't be very efficient when trying to add over one thousand products at a time, I've also been looking at full text search (FREETEXT and contains) but FreeText search seems to bring back alot of results as its matching any and all words in a product description.

Has anyone done something similar in terms of trying to automate which category a product is by its description and can offer some advice or pointers?

Answer 1

So the question as I understand it is, given a description tell me what category this description is applicable to?

A common method to do this kind of work is to build a Naive Bayesian Classification process, and put all of your descriptions through this.

Classification like this usually takes place in two stages.

stage 1 : known description/category pairs are used to "train" the classifier.

stage 2 : Once the classifier is trained, you can then give it unknown data, and it would then return a probability that the description would match a given category.

The classifier in this approach is usually pretty accurate, but given we are dealing with statistics, errors usually do creep in