快速搜索相似的文本

Question

I am supporting a public blog to which users could publish their posts. Some users have more than thousand different texts and they might not remember, that they have already published some text. I would like to help users not to publish duplicates.

Comparing texts for exact equality is not good - user might have changed text a little, or formatting, or copied from a different program, etc. So I need a quick estimate, if there is a similar text in existing database.

My technology stack includes PHP, MySQL and Redis. How can I solve my problem using those or other instruments?

Answer 1

PHP has a function called similar_text which you can use to calculate the amount of matching characters or the similarity in percent.

http://php.net/manual/en/function.similar-text.php

You could then check if the given text is within a certain margin of older blog posts.

If you don't want to check for similarity in text you could try to tag the posts based on tags of the original blog or subject of the blog. And then show the users the posts they made with similar tags.

Answer 2

You can use MySQL's match - against in a full text indexed column.

As an example:

SELECT table.*, 
MATCH(userText) AGAINST ('this is user input') AS relevancy 
FROM table 
ORDER BY relevancy DESC;

So this will give you results ordered by relevancy.

Don't forget to add full text index on column userText .