[英]Python cosine-similarity on all possible pairs in list
I'm learning Python slowly and was wondering if I could have some help. 我正在慢慢学习Python,并且想知道是否可以提供帮助。 I have a list of ips, occurrence_id and vector called info_list : 我有一个ips, occeence_id和向量的列表,称为info_list :
('188.74.64.243', '1', ['0, 1, 1, 0'])
('99.229.98.18', '1', ['0, 1, 1, 1'])
('86.41.253.102', '1', ['1, 1, 1, 1'])
('188.74.64.243', '2', ['0, 1, 1, 0'])
('99.229.98.18', '2', ['0, 1, 1, 1'])
('86.41.253.102', '2', ['1, 1, 1, 1'])
I want to calculate cosine similarity. 我想计算余弦相似度。 I have the following: 我有以下几点:
def cosine_similarity(v1,v2):
sumxx, sumxy, sumyy = 0, 0, 0
for i in range(len(v1)):
x = v1[i]; y = v2[i]
sumxx += x*x
sumyy += y*y
sumxy += x*y
return sumxy/math.sqrt(sumxx*sumyy)
v1 = [0, 1, 1, 0]
v2 = [1, 1, 1, 1]
print(v1, v2, cosine_similarity(v1,v2))
This works great when v1 and v2 are stated. 声明了v1和v2时,这很好用。 My problem is that I'm in a little bit of a loop hole and can't seem to piece together my problem. 我的问题是我陷入了一个小漏洞,似乎无法解决我的问题。 I was hoping for a little help. 我希望能有所帮助。
I need to loop through info_list , taking into consideration each pair of ips that have the same occurrence_id to calculate the cosine_similarity . 我需要遍历info_list,考虑到每对具有相同occurrence_id计算cosine_similarity IPS的。
An example of the output would be a list like so: 输出的示例将是这样的列表:
('188.74.64.243', '99.229.98.18', '1', ['0, 1, 1, 0'],['0, 1, 1, 1'], 0.82 )
('188.74.64.243', '86.41.253.102', '1', ['0, 1, 1, 0'],['1, 1, 1, 1'], 0.70 )
('86.41.253.102', '99.229.98.18', '1', ['0, 1, 1, 1'],['1, 1, 1, 1'], 0.87 )
You can make use of Python's groupby
and combinations
functions as follows: 您可以使用Python的groupby
和combinations
功能如下:
from itertools import groupby, combinations
import math
def cosine_similarity(v1,v2):
sumxx, sumxy, sumyy = 0, 0, 0
for i in range(len(v1)):
x = v1[i]; y = v2[i]
sumxx += x*x
sumyy += y*y
sumxy += x*y
return sumxy/math.sqrt(sumxx * sumyy)
info_list = [
('188.74.64.243', '1', [0, 1, 1, 0]),
('99.229.98.18', '1', [0, 1, 1, 1]),
('86.41.253.102', '1', [1, 1, 1, 1]),
('188.74.64.243', '2', [0, 1, 1, 0]),
('99.229.98.18', '2', [0, 1, 1, 1]),
('86.41.253.102', '2', [1, 1, 1, 1]),
]
for k, g in groupby(info_list, key=lambda x: x[1]):
for x, y in combinations(g, 2):
print (x[0], y[0], x[1], x[2], y[2], cosine_similarity(x[2], y[2]))
print
This will display the following output: 这将显示以下输出:
('188.74.64.243', '99.229.98.18', '1', [0, 1, 1, 0], [0, 1, 1, 1], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '1', [0, 1, 1, 0], [1, 1, 1, 1], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '1', [0, 1, 1, 1], [1, 1, 1, 1], 0.8660254037844387)
('188.74.64.243', '99.229.98.18', '2', [0, 1, 1, 0], [0, 1, 1, 1], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '2', [0, 1, 1, 0], [1, 1, 1, 1], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '2', [0, 1, 1, 1], [1, 1, 1, 1], 0.8660254037844387)
If the list is not sorted, ie the IDs are not grouped together, then the following line could be replaced: 如果列表未排序,即未将ID分组在一起,则可以替换以下行:
for k, g in groupby(sorted(info_list, key=lambda x: x[1]), key=lambda x: x[1]):
Keeping the data as you gave it (with the vector represented by a string), You could write a function which takes two of your tuples, unpacks the string into an int vector, applies the similarity function, and the repackages. 保持数据不变(用字符串表示的向量),可以编写一个函数,该函数接受两个元组,将字符串解压缩为int向量,应用相似性函数,然后重新打包。 Then -- use this function with a basic nested loop: 然后-通过基本的嵌套循环使用此函数:
import math
def cosine_similarity(v1,v2):
sumxx, sumxy, sumyy = 0, 0, 0
for i in range(len(v1)):
x, y = v1[i],v2[i]
sumxx += x*x
sumyy += y*y
sumxy += x*y
return sumxy/math.sqrt(sumxx*sumyy)
def c_sim(t1,t2):
ips1,id1,vlist1 = t1
ips2,id2,vlist2 = t2
v1 = [int(i) for i in vlist1[0].split(',')]
v2 = [int(i) for i in vlist2[0].split(',')]
if id1 == id2:
return ips1,ips2,id1,vlist1,vlist2,cosine_similarity(v1,v2)
def process_list(data_list):
n = len(data_list)
ret_list = []
for i in range(n-1):
for j in range(i+1,n):
t1,t2 = data_list[i],data_list[j]
t = c_sim(t1,t2)
if t: ret_list.append(t)
return ret_list
data = [('188.74.64.243', '1', ['0, 1, 1, 0']),
('99.229.98.18', '1', ['0, 1, 1, 1']),
('86.41.253.102', '1', ['1, 1, 1, 1']),
('188.74.64.243', '2', ['0, 1, 1, 0']),
('99.229.98.18', '2', ['0, 1, 1, 1']),
('86.41.253.102', '2', ['1, 1, 1, 1'])]
for t in process_list(data): print(t)
Output: 输出:
('188.74.64.243', '99.229.98.18', '1', ['0, 1, 1, 0'], ['0, 1, 1, 1'], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '1', ['0, 1, 1, 0'], ['1, 1, 1, 1'], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '1', ['0, 1, 1, 1'], ['1, 1, 1, 1'], 0.8660254037844387)
('188.74.64.243', '99.229.98.18', '2', ['0, 1, 1, 0'], ['0, 1, 1, 1'], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '2', ['0, 1, 1, 0'], ['1, 1, 1, 1'], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '2', ['0, 1, 1, 1'], ['1, 1, 1, 1'], 0.8660254037844387)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.