使用R聚类相似的单词

Question

I am working with RI have two dataframes. The one contains 200000 words like "cat", "cats", "cts", "dogs", and "dog", and the other one contains words like "cat", and "dog".

I want to cluster the first dataframe and replace all the similar words with the word that fits in the second dataframe. For example, "cats" and "cts" to become "cat".

Answer 1

As mentioned by @G5W, the task requires user input. Here is an example of how that could be done

# we have here pre-defined choices: any match must out of 'animals'
animals <- c('cat','dog','mouse')
# here is the text we want to match
text <- c('cats', 'cuts', 'dogs', 'dawg', 'frog', 'lion')
# now we use the string distance metric
# via the package stringdist & using metric 'jw'
# c.f. ?stringdist::stringdist
vapply(seq_along(text), 
       function (k) animals[which.min(stringdist::stringdist(text[k], animals, 'jw'))], 
       character(1))
# [1] "cat" "cat" "dog" "dog" "dog" "dog"

Notice that for instance lion is matched to dog as that is the closest match.

To further illustrate the points made in the comment section, consider the following

stringdist::stringdist('cts', c('cats','cuts'), 'jw')
# [1] 0.08333333 0.08333333

The word cts is equidistant to both cats and cuts . Assume the two word are contained in the matching table animals , then in this case which.min would return (by default) the first instance of the minimal value, so we would obtain cats .

You see how this can turn out to be problematic: assume cts was supposed to be cuts , the above would yield a false value.

Answer 2

Thanks for your question!

I'm currently out of my house and typing the proposed solution on my iPhone, but I'll apply it to your example when I'm home.

The way to convert similar values is by using the agrep function. You don't need any package for it, it's already in R .

Please leave a comment if you need specific examples :)

Here is the functionality:

agrep(pattern, x, max.distance = 0.1, costs = NULL, ignore.case = FALSE, 
      value = FALSE, fixed = TRUE, useBytes = FALSE)