[英]how to efficiently compare two dictionaries of lists of strings using difflib?
I have two large dictionaries of lists.我有两个大型列表字典。 All the elements of the lists are strings.
列表的所有元素都是字符串。 I want to compare all against all and compute their respective similarity - but the naive way I use is obviously very slow and does not scale at all:
我想将所有内容与所有内容进行比较并计算它们各自的相似性 - 但我使用的天真的方式显然非常缓慢并且根本无法扩展:
import numpy as np
import difflib
first_dict = {"first1" : ["aa", "bb","cc", "dd"], "first2" : ["ff", "gg"]}
second_dict = {"second1" : ["cc", "dd", "jj", "aa", "bb"], "second2" : ["ff", "gg"], "second3" : ["hh", "ii"]}
mat = np.empty((len(second_dict), len(first_dict)))
for i, second in enumerate(second_dict.keys()):
for j, first in enumerate(first_dict.keys()):
sm = difflib.SequenceMatcher(None, sorted(first_dict[first]), sorted(second_dict[second]))
mat[i, j] = sm.ratio()
print(mat)
is there a clever way to speed this up?有没有一种聪明的方法可以加快速度?
Your code seems legit.您的代码似乎合法。 I did a couple of little tweaks that would shave off a couple of microseconds per loop:
我做了一些小调整,每个循环可以减少几微秒:
sorted
calls because difflib
can calculated an order-indifferent comparison with quick_ratio
(Checkout the documentation here for the difference between ratio
, quick_ratio
, and real_quick_ratio
).sorted
的调用,因为difflib
可以计算与quick_ratio
的顺序无关的比较(在此处查看文档以了解ratio
、 quick_ratio
和real_quick_ratio
之间的差异)。enumerate
to access mat
by i
and j
. enumerate
不需要通过i
和j
访问mat
。first_dict[index]
and second_dict[index]
first_dict[index]
和second_dict[index]
对列表的访问def naive_ratio_comparison(first_dict, second_dict):
mat = []
for second in second_dict.values():
for first in first_dict.values():
sm = difflib.SequenceMatcher(None, first, second)
mat.append(sm.quick_ratio())
result = np.resize(mat, (len(second_dict), len(first_dict)))
return result
If one dict has M
entries and the other N
, then you're going to have to do M*N
.ratio()
calls.如果一个 dict 有
M
条目,另一个有N
,那么您将不得不进行M*N
.ratio()
调用。 There's no way around that, and it's going to be costly.没有办法解决这个问题,而且代价高昂。
However, you can easily arrange to do only M+N
sorts instead of (as shown) M*N
sorts.但是,您可以轻松地安排只进行
M+N
排序而不是(如图所示) M*N
排序。
For computing .ratio()
, the most valuable hint is in the docs:对于计算
.ratio()
,最有价值的提示在文档中:
SequenceMatcher
computes and caches detailed information about the second sequence, so if you want to compare one sequence against many sequences, useset_seq2()
to set the commonly used sequence once and callset_seq1()
repeatedly, once for each of the other sequences.SequenceMatcher
计算并缓存有关第二个序列的详细信息,因此如果要将一个序列与多个序列进行比较,请使用set_seq2()
设置常用序列一次,然后重复调用set_seq1()
,对其他每个序列执行一次。
Putting that all together:把所有这些放在一起:
firsts = list(map(sorted, first_dict.values())) # sort these only once
sm = difflib.SequenceMatcher(None)
for i, second in enumerate(second_dict.values()):
sm.set_seq2(sorted(second))
for j, first in enumerate(firsts):
sm.set_seq1(first)
mat[i, j] = sm.ratio()
That should deliver exactly the same results.这应该会产生完全相同的结果。 To minimize the number of expensive
.set_seq2()
calls, it would - of course - be best to arrange for the shorter dict to be called "second_dict".为了尽量减少昂贵的
.set_seq2()
调用的数量,当然,最好将较短的 dict 称为“second_dict”。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.