在R中的大数据表中查找字符串的大列表的最快方法

Question

I have a list of around 15.000 user id's,

> head(ID_data)
[1] "A01Y" "AC43" "BBN5" "JK45" "NT66" "WC44"

and a table with 3 columns and around 100.000 rows as a data.table:

> head(USER_data)
              V1                    V2                 V3
1:             0                  John               John
2:          A01Y        Martin 3311290               
3: Peter Johnson            Peter JK45                  x 
4:             1        wc44@email.com     wc44@email.com         
5:            NA                     x          
6:        419223    Christian 21221140     ac43@email.com

I want to know the row index of rows that contain a user id somewhere in one of the 3 columns.

In the example above, the code should find row 2, 3, 4 and 6, since they contain "A01Y", "JK45", "WC44" and "AC43" somewhere in one or more of the 3 columns.

The main problem is the big amount of data.

I have tried pasting "|" between the ID's and use grep to search for "A01Y|JK45" etc.:

toMatch <- paste(ID_data,collapse="|")
V1.matches <- grep(toMatch, USER_data$V1, ignore.case=TRUE)
V2.matches <- grep(toMatch, USER_data$V2, ignore.case=TRUE)
V3.matches <- grep(toMatch, USER_data$V3, ignore.case=TRUE)

but grep can only take a search pattern of around 2.500 ID's, so I would have to go through the ID's in blocks of size 2500. This takes around 15 minutes to compute.

I have also tried using strapplyc, which can take a search pattern of around 9.999 ID's.

Is there a faster way to find the row indices?

I was thinking of using sqldf() and do something like

sqldf("SELECT * FROM USER_data, ID_data WHERE USER_data LIKE '%'+ID_data+'%'")

but I'm not sure how to do this exactly.

Thanks a lot in advance for any suggestions.

Answer 1

Not sure if this fast enough, but I've done it before with many rows and IDs. It took me some time, but no need to process IDs in blocks.

# list of ids
IDs = c("A01Y", "AC43", "BBN5", "JK45", "NT66", "WC44")

# example dataframe
dt =  data.frame(V1 = c("Christian 21223456","x", "wc44@email.com"),
                 V2 = c("0 John","1 wc44@email.com",  "wc44@email.com"),
                 V3 = c("1","0","A01Y Martin 3311290"))

dt

#                   V1               V2                  V3
# 1 Christian 21223456           0 John                   1
# 2                  x 1 wc44@email.com                   0
# 3     wc44@email.com   wc44@email.com A01Y Martin 3311290


# combine row elements in one big string
dt_rows = apply(dt, 1, function(x) paste(x,collapse = " "))

# update to lower case
IDs = tolower(IDs)
dt_rows = tolower(dt_rows)

# find in which row you have matches
sapply(IDs, grepl, dt_rows) 

#       a01y  ac43  bbn5  jk45  nt66  wc44
# [1,] FALSE FALSE FALSE FALSE FALSE FALSE
# [2,] FALSE FALSE FALSE FALSE FALSE  TRUE
# [3,]  TRUE FALSE FALSE FALSE FALSE  TRUE


# find which row id has a match (at least one match)
which(apply(sapply(IDs, grepl, dt_rows), 1, sum) >= 1) 

# [1] 2 3