[英]Python3 - Grouping similar strings together
我想要做的是將一個小說網站上的字符串組合在一起。 帖子的標題通常采用以下格式:
titles = ['Series Name: Part 1 - This is the chapter name',
'[OC] Series Name - Part 2 - Another name with the word chapter and extra oc at the start',
"[OC] Series Name = part 3 = punctuation could be not matching, so we can't always trust common substrings",
'{OC} Another cool story - Part I - This is the chapter name',
'{OC} another cool story: part II: another post title',
'{OC} another cool story part III but the author forgot delimiters',
"this is a one-off story, so it doesn't have any friends"]
分隔符等並不總是存在,並且可能會有一些變化。
我首先將字符串規范化為字母數字字符。
import re
from pprint import pprint as pp
titles = [] # from above
normalized = []
for title in titles:
title = re.sub(r'\bOC\b', '', title)
title = re.sub(r'[^a-zA-Z0-9\']+', ' ', title)
title = title.strip()
normalized.append(title)
pp(normalized)
這使
['Series Name Part 1 This is the chapter name',
'Series Name Part 2 Another name with the word chapter and extra oc at the start',
"Series Name part 3 punctuation could be not matching so we can't always trust common substrings",
'Another cool story Part I This is the chapter name',
'another cool story part II another post title',
'another cool story part III but the author forgot delimiters',
"this is a one off story so it doesn't have any friends"]
我希望的 output 是:
['Series Name',
'Another cool story',
"this is a one-off story, so it doesn't have any friends"] # last element optional
我知道比較字符串的幾種不同方法......
difflib.SequenceMatcher.ratio()
我也聽說過 Jaro-Winkler 和 FuzzyWuzzy。
但真正重要的是我們可以得到一個數字來顯示字符串之間的相似性。
我想我需要想出(大部分)一個二維矩陣來比較每個字符串。 但是一旦我知道了,我就無法思考如何將它們實際分成組。
我發現另一個似乎已經完成了第一部分的帖子......但是我不確定如何從那里繼續。
scipy.cluster 起初看起來很有希望……但后來我就不知所措了。
另一個想法是以某種方式將itertools.combinations()與functools.reduce()與上述距離度量之一結合起來。
我是不是想太多了? 看起來這應該很簡單,但它在我的腦海中沒有意義。
這是CKM回答中提出的想法的實現: https://stackoverflow.com/a/61671971/42346
首先去掉標點符號——這對你的目的並不重要——使用這個答案: https://stackoverflow.com/a/15555162/42346
然后,我們將使用此處描述的技術之一: https://blog.eduonix.com/artificial-intelligence/clustering-similar-sentences-together-using-machine-learning/來聚類相似的句子。
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w+') # only alphanumeric characters
lol_tokenized = []
for title in titles:
lol_tokenized.append(tokenizer.tokenize(title))
然后獲取標題的數字表示:
import numpy as np
from gensim.models import Word2Vec
m = Word2Vec(lol_tokenized,size=50,min_count=1,cbow_mean=1)
def vectorizer(sent,m):
vec = []
numw = 0
for w in sent:
try:
if numw == 0:
vec = m[w]
else:
vec = np.add(vec, m[w])
numw += 1
except Exception as e:
print(e)
return np.asarray(vec) / numw
l = []
for i in lol_tokenized:
l.append(vectorizer(i,m))
X = np.array(l)
天哪,那是很多。
現在你必須進行聚類。
from sklearn.cluster import KMeans
clf = KMeans(n_clusters=2,init='k-means++',n_init=100,random_state=0)
labels = clf.fit_predict(X)
print(labels)
for index, sentence in enumerate(lol_tokenized):
print(str(labels[index]) + ":" + str(sentence))
[1 1 0 1 0 0 0]
1:['Series', 'Name', 'Part', '1', 'This', 'is', 'the', 'chapter', 'name']
1:['OC', 'Series', 'Name', 'Part', '2', 'Another', 'name', 'with', 'the', 'word', 'chapter', 'and', 'extra', 'oc', 'at', 'the', 'start']
0:['OC', 'Series', 'Name', 'part', '3', 'punctuation', 'could', 'be', 'not', 'matching', 'so', 'we', 'can', 't', 'always', 'trust', 'common', 'substrings']
1:['OC', 'Another', 'cool', 'story', 'Part', 'I', 'This', 'is', 'the', 'chapter', 'name']
0:['OC', 'another', 'cool', 'story', 'part', 'II', 'another', 'post', 'title']
0:['OC', 'another', 'cool', 'story', 'part', 'III', 'but', 'the', 'author', 'forgot', 'delimiters']
0:['this', 'is', 'a', 'one', 'off', 'story', 'so', 'it', 'doesn', 't', 'have', 'any', 'friends']
然后你可以拉出索引 == 1 的那些:
for index, sentence in enumerate(lol_tokenized):
if labels[index] == 1:
print(sentence)
['Series', 'Name', 'Part', '1', 'This', 'is', 'the', 'chapter', 'name']
['OC', 'Series', 'Name', 'Part', '2', 'Another', 'name', 'with', 'the', 'word', 'chapter', 'and', 'extra', 'oc', 'at', 'the', 'start']
['OC', 'Another', 'cool', 'story', 'Part', 'I', 'This', 'is', 'the', 'chapter', 'name']
您的任務屬於所謂的semantic similarity
。 我建議你進行如下操作:
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.