[英]Slow loop R, how make it faster?
I have a list of e-mails and I would like to compare patterns (similarity) among the rows using longest common substring to compare them. 我有一封电子邮件列表,我想使用最长的公共子字符串比较行之间的模式(相似性)。
data is a data frame with e-mails: 数据是带有电子邮件的数据框:
V1
1 "01003@163.com"
2 "cloud@coldmail.com"
3 "den_smukk_kiilar@hotmail.com"
4 "Esteban.verduzco@gmail.com"
5 "freiheitmensch@gmail.com"
6 "mitsoanastos@yahoo.com"
7 "ahmedsir744@yahoo.com"
8 ...
This is my code: 这是我的代码:
library(stringdist)
for(i in 1:nrow(data)) {
sample <- data[i,]
for(j in (i+1):nrow(data)) if(i+1 <= nrow(data)) {
if((stringdist(data[j,],sample,method='lcs'))<=3) { #number of different characteres 3 (123.456 == 123.321)
duplicate <- data[j,]
email1 = as.character(data[i,])
email2 = as.character(data[j,])
pair <- cbind(email1, email2)
output3[dfrow, ] <- pair
dfrow <- dfrow + 1
}
}
}
and the "outupt" is a data frame showing the similar e-mails. “ outupt”是显示类似电子邮件的数据框。
email1 email2
1 "01079@163.com" "01069@163.com"
I have 300k e-mails, this will take forever... 我有30万封电子邮件,这将永远需要...
Is there a better way to do it? 有更好的方法吗?
Thanks! 谢谢!
Here's an attempt: 这是一个尝试:
library(stringdist)
library(stringi)
library(dplyr)
library(tidyr)
# Hypothetical data frame
data <- data.frame(V1 = paste0(stri_rand_strings(5, 3, "[a-z]"),
"@", stri_rand_strings(5, 2, "[a-z]"), ".com"),
stringsAsFactors = FALSE)
Basically you create a string distance pairwise matrix, wrap it in a data frame, replace all string distances that are equal or less than 3 with the corresponding V1
value and the rest with NA
. 基本上,您将创建一个字符串距离成对矩阵,将其包装在数据框中,将等于或小于3的所有字符串距离替换为相应的
V1
值,其余的替换为NA
。 Then, you remove the now unnecessary V1
column, gather()
the data in a tidy format and remove NA
s. 然后,删除现在不再需要的
V1
列,以整齐的格式gather()
数据并删除NA
。
data %>%
data.frame(stringdistmatrix(.$V1, .$V1, useNames = TRUE, method = "lcs"),
row.names = NULL) %>%
# V1 wnw.fa.com kty.hm.com brs.wk.com pib.uo.com ryu.iq.com
#1 wnw@fa.com 0 10 10 10 10
#2 kty@hm.com 10 0 10 10 8
#3 brs@wk.com 10 10 0 8 8
#4 pib@uo.com 10 10 8 0 10
#5 ryu@iq.com 10 8 8 10 0
# here you need to replace '8' by '3' for your example
mutate_each(funs(ifelse(. <= 8 & . != 0, V1, NA)), -V1) %>%
# V1 wnw.fa.com kty.hm.com brs.wk.com pib.uo.com ryu.iq.com
#1 wnw@fa.com NA <NA> <NA> <NA> <NA>
#2 kty@hm.com NA <NA> <NA> <NA> kty@hm.com
#3 brs@wk.com NA <NA> <NA> brs@wk.com brs@wk.com
#4 pib@uo.com NA <NA> pib@uo.com <NA> <NA>
#5 ryu@iq.com NA ryu@iq.com ryu@iq.com <NA> <NA>
select(-V1) %>%
gather(email1, email2) %>%
na.omit() %>%
mutate(email1 = stri_replace_first(email1, fixed = ".", "@"))
Which gives: 这使:
# email1 email2
#1 kty@hm.com ryu@iq.com
#2 brs@wk.com pib@uo.com
#3 brs@wk.com ryu@iq.com
#4 pib@uo.com brs@wk.com
#5 ryu@iq.com kty@hm.com
#6 ryu@iq.com brs@wk.com
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.